Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
2026-09-26T01:55:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:55:23Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260926 01:55:23.639014 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.190:44589
--webserver_interface=127.8.76.190
--webserver_port=0
--builtin_ntp_servers=127.8.76.148:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260926 01:55:23.962000 8507 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:55:23.962723 8507 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:55:23.963140 8507 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:55:23.998263 8507 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:55:23.998605 8507 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:55:23.998824 8507 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:55:23.999037 8507 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:55:24.038803 8507 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.190:44589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.76.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:55:24.040397 8507 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:55:24.042258 8507 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:55:24.056859 8513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:24.057514 8514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:24.061259 8516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:25.208633 8515 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1146 milliseconds
I20260926 01:55:25.208751 8507 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:55:25.210126 8507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:55:25.213367 8507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:55:25.214824 8507 hybrid_clock.cc:648] HybridClock initialized: now 1790387725214766 us; error 81 us; skew 500 ppm
I20260926 01:55:25.215740 8507 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:55:25.223347 8507 webserver.cc:533] Webserver started at http://127.8.76.190:46443/ using document root <none> and password file <none>
I20260926 01:55:25.224426 8507 fs_manager.cc:362] Metadata directory not provided
I20260926 01:55:25.224660 8507 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:55:25.225171 8507 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:55:25.230813 8507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data/instance:
uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
format_stamp: "Formatted at 2026-09-26 01:55:25 on dist-test-slave-5pk1"
I20260926 01:55:25.232030 8507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/wal/instance:
uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
format_stamp: "Formatted at 2026-09-26 01:55:25 on dist-test-slave-5pk1"
I20260926 01:55:25.239931 8507 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 01:55:25.246663 8523 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:25.247869 8507 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 01:55:25.248242 8507 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/wal
uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
format_stamp: "Formatted at 2026-09-26 01:55:25 on dist-test-slave-5pk1"
I20260926 01:55:25.248612 8507 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:55:25.309998 8507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:55:25.312180 8507 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:55:25.312718 8507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:55:25.400960 8507 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.190:44589
I20260926 01:55:25.401022 8574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.190:44589 every 8 connection(s)
I20260926 01:55:25.404029 8507 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
I20260926 01:55:25.411813 8575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:25.414973 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 8507
I20260926 01:55:25.415427 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-0/wal/instance
I20260926 01:55:25.419209 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.189:34163
--webserver_interface=127.8.76.189
--webserver_port=0
--builtin_ntp_servers=127.8.76.148:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260926 01:55:25.423892 8575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:55:25.455050 8575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
W20260926 01:55:25.458841 8526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
W20260926 01:55:25.463264 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34163: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
I20260926 01:55:25.512744 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } attempt: 1
W20260926 01:55:25.517606 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34163: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
I20260926 01:55:25.586133 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } attempt: 2
W20260926 01:55:25.591693 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34163: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
I20260926 01:55:25.683220 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } attempt: 3
W20260926 01:55:25.688839 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34163: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
W20260926 01:55:25.772450 8577 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:55:25.773101 8577 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:55:25.773492 8577 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:55:25.808495 8577 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:55:25.808837 8577 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:55:25.809053 8577 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:55:25.809325 8577 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:55:25.832368 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } attempt: 4
W20260926 01:55:25.837390 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34163: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
I20260926 01:55:25.851027 8577 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.189:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.76.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:55:25.852708 8577 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:55:25.854712 8577 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:55:25.869798 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:26.136960 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } attempt: 5
W20260926 01:55:26.144634 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34163: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
I20260926 01:55:26.692226 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } attempt: 6
W20260926 01:55:26.721042 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34163: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34163: connect: Connection refused (error 111)
W20260926 01:55:25.869992 8587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:27.220363 8589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:27.223624 8588 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1347 milliseconds
W20260926 01:55:27.224309 8577 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.354s user 0.484s sys 0.865s
W20260926 01:55:27.224617 8577 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.355s user 0.484s sys 0.865s
I20260926 01:55:27.224846 8577 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:55:27.226058 8577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:55:27.228961 8577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:55:27.232681 8577 hybrid_clock.cc:648] HybridClock initialized: now 1790387727232633 us; error 55 us; skew 500 ppm
I20260926 01:55:27.233655 8577 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:55:27.242805 8577 webserver.cc:533] Webserver started at http://127.8.76.189:40571/ using document root <none> and password file <none>
I20260926 01:55:27.243868 8577 fs_manager.cc:362] Metadata directory not provided
I20260926 01:55:27.244100 8577 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:55:27.244596 8577 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:55:27.249327 8577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data/instance:
uuid: "9ef98e01cb9a44408ebf207163f208b5"
format_stamp: "Formatted at 2026-09-26 01:55:27 on dist-test-slave-5pk1"
I20260926 01:55:27.250551 8577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/wal/instance:
uuid: "9ef98e01cb9a44408ebf207163f208b5"
format_stamp: "Formatted at 2026-09-26 01:55:27 on dist-test-slave-5pk1"
I20260926 01:55:27.260072 8577 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20260926 01:55:27.267349 8598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:27.268724 8577 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 01:55:27.269073 8577 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/wal
uuid: "9ef98e01cb9a44408ebf207163f208b5"
format_stamp: "Formatted at 2026-09-26 01:55:27 on dist-test-slave-5pk1"
I20260926 01:55:27.269426 8577 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:55:27.361374 8577 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:55:27.363190 8577 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:55:27.363703 8577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:55:27.458765 8577 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.189:34163
I20260926 01:55:27.458822 8649 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.189:34163 every 8 connection(s)
I20260926 01:55:27.461763 8577 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/data/info.pb
I20260926 01:55:27.469188 8650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:27.469156 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 8577
I20260926 01:55:27.469674 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-1/wal/instance
I20260926 01:55:27.473321 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:38143
--webserver_interface=127.8.76.188
--webserver_port=0
--builtin_ntp_servers=127.8.76.148:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260926 01:55:27.485354 8650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:55:27.518865 8650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:55:27.535662 8650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } has no permanent_uuid. Determining permanent_uuid...
W20260926 01:55:27.538285 8599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
W20260926 01:55:27.542982 8650 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:27.592701 8650 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 1
W20260926 01:55:27.597806 8650 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:27.666229 8650 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 2
W20260926 01:55:27.670905 8650 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:27.762292 8650 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 3
W20260926 01:55:27.766937 8650 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:27.781581 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } attempt: 7
I20260926 01:55:27.796654 8575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } has no permanent_uuid. Determining permanent_uuid...
W20260926 01:55:27.803509 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
W20260926 01:55:27.834371 8652 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:55:27.835040 8652 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:55:27.835440 8652 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260926 01:55:27.862068 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 1
W20260926 01:55:27.866914 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
W20260926 01:55:27.871554 8652 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:55:27.871855 8652 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:55:27.872068 8652 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:55:27.872277 8652 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:55:27.910353 8650 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 4
I20260926 01:55:27.913420 8652 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:38143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.76.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
W20260926 01:55:27.915447 8650 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:27.915408 8652 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:55:27.917333 8652 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:55:27.929731 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:27.948866 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 2
W20260926 01:55:27.956173 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:28.041738 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 3
W20260926 01:55:28.049778 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:28.190289 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 4
W20260926 01:55:28.198448 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:28.214847 8650 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 5
W20260926 01:55:28.222550 8650 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:28.481992 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 5
W20260926 01:55:28.490304 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:28.770083 8650 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 6
W20260926 01:55:28.794654 8650 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:29.042852 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 6
W20260926 01:55:29.052994 8575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
W20260926 01:55:27.931185 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:29.297942 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:29.300678 8652 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.370s user 0.450s sys 0.895s
W20260926 01:55:29.301083 8652 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.371s user 0.450s sys 0.895s
W20260926 01:55:29.302062 8666 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1364 milliseconds
I20260926 01:55:29.302021 8652 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:55:29.303908 8652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:55:29.308295 8652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:55:29.309829 8652 hybrid_clock.cc:648] HybridClock initialized: now 1790387729309793 us; error 40 us; skew 500 ppm
I20260926 01:55:29.310819 8652 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:55:29.320391 8652 webserver.cc:533] Webserver started at http://127.8.76.188:33837/ using document root <none> and password file <none>
I20260926 01:55:29.321437 8652 fs_manager.cc:362] Metadata directory not provided
I20260926 01:55:29.321666 8652 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:55:29.322165 8652 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:55:29.327299 8652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/instance:
uuid: "f75ba46482de422ea23daf17aa100849"
format_stamp: "Formatted at 2026-09-26 01:55:29 on dist-test-slave-5pk1"
I20260926 01:55:29.328871 8652 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal/instance:
uuid: "f75ba46482de422ea23daf17aa100849"
format_stamp: "Formatted at 2026-09-26 01:55:29 on dist-test-slave-5pk1"
I20260926 01:55:29.338333 8652 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260926 01:55:29.346040 8678 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:29.347599 8652 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 01:55:29.347965 8652 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
uuid: "f75ba46482de422ea23daf17aa100849"
format_stamp: "Formatted at 2026-09-26 01:55:29 on dist-test-slave-5pk1"
I20260926 01:55:29.348337 8652 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:55:29.429560 8652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:55:29.431420 8652 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:55:29.431897 8652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:55:29.527998 8652 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.188:38143
I20260926 01:55:29.528046 8729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.188:38143 every 8 connection(s)
I20260926 01:55:29.531152 8652 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
I20260926 01:55:29.536192 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 8652
I20260926 01:55:29.536610 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal/instance
I20260926 01:55:29.538445 8730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:29.558499 8730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:55:29.586838 8730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:55:29.598948 8730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:55:29.620056 8730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Bootstrap starting.
I20260926 01:55:29.629388 8730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:29.632012 8730 log.cc:826] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Log is configured to *not* fsync() on all Append() calls
I20260926 01:55:29.638222 8730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: No bootstrap required, opened a new log
I20260926 01:55:29.661657 8730 raft_consensus.cc:359] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:29.662364 8730 raft_consensus.cc:385] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:29.662672 8730 raft_consensus.cc:740] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75ba46482de422ea23daf17aa100849, State: Initialized, Role: FOLLOWER
I20260926 01:55:29.663508 8730 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:29.665964 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:29.666834 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:29.668125 8730 sys_catalog.cc:565] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 01:55:29.689851 8684 tablet.cc:2433] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:55:29.700616 8750 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:55:29.700925 8750 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:55:29.696683 8652 master_runner.cc:187] Error getting master registration for 127.8.76.190:44589: OK, instance_id { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" instance_seqno: 1790387725376559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:55:29.729195 8652 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:55:29.729808 8739 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:29.730401 8739 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:29.734733 8739 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:29.855193 8650 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 7
I20260926 01:55:29.876977 8650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Bootstrap starting.
I20260926 01:55:29.885656 8650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:29.888314 8650 log.cc:826] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Log is configured to *not* fsync() on all Append() calls
I20260926 01:55:29.894254 8650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: No bootstrap required, opened a new log
I20260926 01:55:29.925622 8650 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:29.926643 8650 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:29.926978 8650 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef98e01cb9a44408ebf207163f208b5, State: Initialized, Role: FOLLOWER
I20260926 01:55:29.927929 8650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:29.931504 8754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:29.932307 8754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:29.933344 8650 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:55:29.933645 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:29.935729 8625 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 0.
I20260926 01:55:29.937737 8681 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters:
I20260926 01:55:29.938875 8739 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:29.939246 8739 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:29.939590 8739 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:29.955608 8739 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:29.962152 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:55:29.962810 8625 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:29.970785 8739 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 1 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:29.972004 8625 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 1.
I20260926 01:55:29.974094 8681 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters:
I20260926 01:55:29.975034 8739 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:29.983393 8739 raft_consensus.cc:697] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 1 LEADER]: Becoming Leader. State: Replica: f75ba46482de422ea23daf17aa100849, State: Running, Role: LEADER
I20260926 01:55:29.984918 8739 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:30.011533 8763 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75ba46482de422ea23daf17aa100849. Latest consensus state: current_term: 1 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.012367 8763 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:30.017372 8768 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:55:30.026106 8768 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:55:30.057966 8625 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Refusing update from remote peer f75ba46482de422ea23daf17aa100849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:55:30.061812 8739 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:55:30.062934 8767 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:55:30.063299 8767 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:55:30.086529 8575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } attempt: 7
I20260926 01:55:30.008054 8577 master_runner.cc:187] Error getting master registration for 127.8.76.190:44589: OK, instance_id { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" instance_seqno: 1790387725376559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:55:30.099180 8577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:55:30.123594 8604 tablet.cc:2433] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:55:30.129899 8754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75ba46482de422ea23daf17aa100849. Latest consensus state: current_term: 1 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.135741 8754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:30.141496 8770 mvcc.cc:204] Tried to move back new op lower bound from 7333428142271070208 to 7333428142068334592. Current Snapshot: MvccSnapshot[applied={T|T < 7333428142271070208}]
I20260926 01:55:30.145884 8575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Bootstrap starting.
I20260926 01:55:30.146751 8772 mvcc.cc:204] Tried to move back new op lower bound from 7333428142271070208 to 7333428142068334592. Current Snapshot: MvccSnapshot[applied={T|T < 7333428142271070208}]
I20260926 01:55:30.155603 8575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:30.155943 8754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.156670 8754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:30.158540 8575 log.cc:826] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Log is configured to *not* fsync() on all Append() calls
I20260926 01:55:30.170085 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.170871 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:30.175753 8575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: No bootstrap required, opened a new log
I20260926 01:55:30.178190 8768 catalog_manager.cc:1396] Generated new cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:55:30.178606 8768 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:55:30.225924 8575 raft_consensus.cc:359] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:30.227128 8575 raft_consensus.cc:385] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:30.229401 8575 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Initialized, Role: FOLLOWER
I20260926 01:55:30.230643 8575 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:30.232707 8768 catalog_manager.cc:1419] Generated new certificate authority record
I20260926 01:55:30.235034 8768 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:55:30.266170 8575 sys_catalog.cc:565] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:55:30.266323 8549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
I20260926 01:55:30.267231 8549 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:30.271159 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:55:30.272318 8550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44142 (request call id 2) took 537 ms (client timeout 640 ms). Trace:
W20260926 01:55:30.272691 8550 rpcz_store.cc:269] 0926 01:55:29.734839 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:29.735015 (+ 176us) service_pool.cc:224] Handling call
0926 01:55:30.272223 (+537208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:30.264770 8778 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.278735 8778 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:30.298861 8549 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 1.
I20260926 01:55:30.304703 8548 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Refusing update from remote peer f75ba46482de422ea23daf17aa100849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:55:30.315052 8771 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:55:30.338475 8529 tablet.cc:2433] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:55:30.346716 8768 catalog_manager.cc:6219] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Generated new TSK 0
I20260926 01:55:30.348061 8768 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:55:30.377888 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.76.129:0
--local_ip_for_outbound_sockets=127.8.76.129
--webserver_interface=127.8.76.129
--webserver_port=0
--tserver_master_addrs=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--builtin_ntp_servers=127.8.76.148:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260926 01:55:30.417287 8778 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75ba46482de422ea23daf17aa100849. Latest consensus state: current_term: 1 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.418098 8778 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:30.418136 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.418922 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:30.419832 8789 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:30.421713 8789 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:30.472158 8794 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:55:30.472462 8794 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: loading cluster ID for follower catalog manager: success
I20260926 01:55:30.496759 8794 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: acquiring CA information for follower catalog manager: success
I20260926 01:55:30.502912 8794 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:55:30.861544 8796 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:30.868666 8796 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:30.872969 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:30.873687 8796 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:30.875824 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:30.877842 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:30.879267 8796 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:55:30.968775 8797 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:30.973419 8797 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:30.977670 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:30.985844 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:30.987756 8524 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba; no voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849
I20260926 01:55:30.995211 8797 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:30.995397 8800 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:55:31.071916 8767 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:55:31.072224 8767 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: loading cluster ID for follower catalog manager: success
I20260926 01:55:31.087453 8767 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: acquiring CA information for follower catalog manager: success
I20260926 01:55:31.099707 8767 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 01:55:31.176819 8791 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:55:31.177740 8791 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:55:31.178771 8791 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:55:31.243860 8791 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:55:31.245625 8791 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.76.129
I20260926 01:55:31.315778 8791 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.76.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.76.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.76.129
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:55:31.318140 8791 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:55:31.320948 8791 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:55:31.359301 8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:31.887454 8812 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:31.888329 8812 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:31.891913 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:31.901360 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:31.903565 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:31.908855 8812 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:31.910135 8812 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:55:31.960906 8814 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:31.961651 8814 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:31.963937 8814 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:31.965499 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:31.968873 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:31.969650 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 1.
I20260926 01:55:31.971287 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters: f75ba46482de422ea23daf17aa100849
I20260926 01:55:31.972210 8814 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:55:31.972545 8814 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Starting leader election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:31.972872 8814 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:55:31.981257 8814 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:31.991600 8814 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 2 election: Requested vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:31.992871 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:55:31.993518 8624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:55:32.001972 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 2.
I20260926 01:55:32.008563 8625 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Rejecting Update request from peer f75ba46482de422ea23daf17aa100849 for earlier term 1. Current term is 2. Ops: []
I20260926 01:55:32.020569 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:55:32.021687 8814 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:55:32.022605 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849"
I20260926 01:55:32.021252 8816 consensus_queue.cc:1059] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260926 01:55:32.037099 8814 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 2 LEADER]: Becoming Leader. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:55:32.038021 8815 raft_consensus.cc:3055] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 1 LEADER]: Stepping down as leader of term 1
I20260926 01:55:32.038527 8815 raft_consensus.cc:740] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f75ba46482de422ea23daf17aa100849, State: Running, Role: LEADER
I20260926 01:55:32.038400 8814 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:32.040324 8815 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:32.041690 8815 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:55:32.059974 8817 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 2 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:32.061033 8817 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:32.066824 8820 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:55:32.085716 8820 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:55:32.090517 8820 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:55:32.102399 8820 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:55:32.110435 8820 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:55:32.116978 8820 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loaded TSK: 0
I20260926 01:55:32.123903 8820 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:55:32.544726 8624 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260926 01:55:32.548823 8814 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260926 01:55:32.552815 8705 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260926 01:55:32.554039 8815 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:32.554703 8815 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:32.554944 8817 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260926 01:55:32.556918 8815 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:32.574880 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:32.576776 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:32.579622 8679 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:55:32.599623 8815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 2 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:32.600407 8815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:32.600720 8823 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260926 01:55:32.621476 8814 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:32.622308 8814 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:32.631208 8823 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:32.635286 8823 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:32.636536 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 2 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:32.637391 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:32.653264 8817 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:32.654673 8817 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:32.670171 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:32.672366 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:32.708412 8750 catalog_manager.cc:1651] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:55:33.108237 8823 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:33.108946 8823 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:33.113308 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:33.113938 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:33.116727 8681 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:55:33.117836 8823 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:33.118443 8823 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260926 01:55:33.635648 8831 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:33.636978 8831 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:33.640106 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:33.643572 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:33.645500 8681 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:55:33.651005 8831 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:33.652949 8832 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260926 01:55:32.748594 8805 debug-util.cc:398] Leaking SignalData structure 0x7b0800037180 after lost signal to thread 8791
W20260926 01:55:33.945290 8805 kernel_stack_watchdog.cc:198] Thread 8791 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 383ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260926 01:55:31.363034 8807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:33.947750 8791 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.585s user 0.489s sys 1.022s
W20260926 01:55:33.948148 8791 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.585s user 0.489s sys 1.022s
W20260926 01:55:33.948788 8810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:33.952673 8808 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2587 milliseconds
I20260926 01:55:33.952760 8791 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:55:33.954016 8791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:55:33.956854 8791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:55:33.958288 8791 hybrid_clock.cc:648] HybridClock initialized: now 1790387733958284 us; error 70 us; skew 500 ppm
I20260926 01:55:33.959178 8791 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:55:33.968044 8791 webserver.cc:533] Webserver started at http://127.8.76.129:39371/ using document root <none> and password file <none>
I20260926 01:55:33.969530 8791 fs_manager.cc:362] Metadata directory not provided
I20260926 01:55:33.969837 8791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:55:33.970521 8791 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:55:33.976596 8837 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:33.977034 8837 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:33.977448 8791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data/instance:
uuid: "49bd320bf73f4252804ec867ec077510"
format_stamp: "Formatted at 2026-09-26 01:55:33 on dist-test-slave-5pk1"
I20260926 01:55:33.978791 8837 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:33.979216 8791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/instance:
uuid: "49bd320bf73f4252804ec867ec077510"
format_stamp: "Formatted at 2026-09-26 01:55:33 on dist-test-slave-5pk1"
I20260926 01:55:33.980381 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:33.981222 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:33.983487 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:33.984126 8837 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260926 01:55:33.990120 8791 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s
I20260926 01:55:33.998665 8839 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:34.000207 8791 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 01:55:34.000689 8791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal
uuid: "49bd320bf73f4252804ec867ec077510"
format_stamp: "Formatted at 2026-09-26 01:55:33 on dist-test-slave-5pk1"
I20260926 01:55:34.001221 8791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:55:34.070670 8791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:55:34.072950 8791 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:55:34.073559 8791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:55:34.077577 8791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260926 01:55:34.084127 8791 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:55:34.084450 8791 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:34.084806 8791 ts_tablet_manager.cc:616] Registered 0 tablets
I20260926 01:55:34.085029 8791 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:34.279049 8791 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.129:43257
I20260926 01:55:34.279842 8951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.129:43257 every 8 connection(s)
I20260926 01:55:34.283309 8791 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/data/info.pb
I20260926 01:55:34.289803 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 8791
I20260926 01:55:34.290580 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/instance
I20260926 01:55:34.313524 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.76.130:0
--local_ip_for_outbound_sockets=127.8.76.130
--webserver_interface=127.8.76.130
--webserver_port=0
--tserver_master_addrs=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--builtin_ntp_servers=127.8.76.148:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260926 01:55:34.342268 8952 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:55:34.342885 8952 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:34.344483 8952 heartbeater.cc:507] Master 127.8.76.188:38143 requested a full tablet report, sending...
I20260926 01:55:34.349048 8695 ts_manager.cc:194] Registered new tserver with Master: 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:34.357352 8954 heartbeater.cc:344] Connected to a master server at 127.8.76.190:44589
I20260926 01:55:34.357743 8954 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:34.358018 8953 heartbeater.cc:344] Connected to a master server at 127.8.76.189:34163
I20260926 01:55:34.358342 8953 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:34.358803 8954 heartbeater.cc:507] Master 127.8.76.190:44589 requested a full tablet report, sending...
I20260926 01:55:34.359324 8953 heartbeater.cc:507] Master 127.8.76.189:34163 requested a full tablet report, sending...
I20260926 01:55:34.362247 8540 ts_manager.cc:194] Registered new tserver with Master: 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:34.363889 8615 ts_manager.cc:194] Registered new tserver with Master: 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:34.366770 8540 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.76.129:35729
W20260926 01:55:34.557516 8571 debug-util.cc:398] Leaking SignalData structure 0x7b08000975a0 after lost signal to thread 8508
I20260926 01:55:34.578193 8963 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:34.578926 8963 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:34.583004 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:34.584721 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:34.585352 8963 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:34.586390 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:34.588267 8963 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260926 01:55:34.667384 8964 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:34.668165 8964 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:34.671634 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:34.675220 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:34.674997 8964 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:34.676528 8625 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 2.
I20260926 01:55:34.677976 8681 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters: ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:55:34.679404 8964 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 01:55:34.679936 8964 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:34.680547 8964 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:55:34.688719 8964 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:34.693192 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
I20260926 01:55:34.700232 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:55:34.701995 8681 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:55:34.707399 8964 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 3 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:34.709197 8964 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260926 01:55:34.771733 8705 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Rejecting Update request from peer ca25983598bc4c0688340d8e7bc0f6ba for earlier term 2. Current term is 3. Ops: []
I20260926 01:55:34.773350 8965 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260926 01:55:34.774130 8962 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:55:34.774838 8962 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:55:34.775856 8962 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260926 01:55:34.775517 8971 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 2 LEADER]: Stepping down as leader of term 2
I20260926 01:55:34.776829 8971 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:55:34.777640 8971 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:34.778992 8971 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 2 FOLLOWER]: Advancing to term 3
W20260926 01:55:34.816924 8962 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:55:34.817991 8962 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.76.130
I20260926 01:55:34.864346 8962 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.76.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.76.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.76.130
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:55:34.865995 8962 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:55:34.868400 8962 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260926 01:55:34.871585 8954 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
W20260926 01:55:34.886283 8973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:35.121002 8577 master_runner.cc:264] No leader master found from master 9ef98e01cb9a44408ebf207163f208b5
I20260926 01:55:35.121344 8577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:55:35.216584 8979 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:35.218147 8979 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:35.221837 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:35.222605 8550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef98e01cb9a44408ebf207163f208b5 in term 3.
I20260926 01:55:35.224381 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:55:35.237470 8979 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:35.240898 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:35.241839 8705 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ef98e01cb9a44408ebf207163f208b5 in current term 3: Already voted for candidate f75ba46482de422ea23daf17aa100849 in this term.
I20260926 01:55:35.239022 8980 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 01:55:35.244419 8980 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:35.244949 8980 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:55:35.255357 8980 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:35.266480 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
I20260926 01:55:35.297313 8550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef98e01cb9a44408ebf207163f208b5 in term 3.
I20260926 01:55:35.301460 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:55:35.302744 8979 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:55:35.317984 8979 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 3 LEADER]: Becoming Leader. State: Replica: 9ef98e01cb9a44408ebf207163f208b5, State: Running, Role: LEADER
I20260926 01:55:35.319481 8979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:35.327178 8981 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:35.327814 8981 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:35.333467 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849"
I20260926 01:55:35.334375 8705 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ef98e01cb9a44408ebf207163f208b5 in current term 3: Already voted for candidate f75ba46482de422ea23daf17aa100849 in this term.
I20260926 01:55:35.337579 8980 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 3 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:35.344034 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef98e01cb9a44408ebf207163f208b5. Latest consensus state: current_term: 3 leader_uuid: "9ef98e01cb9a44408ebf207163f208b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:35.345232 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:35.348536 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:35.352247 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:35.353642 8550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 3.
I20260926 01:55:35.356027 8679 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5
I20260926 01:55:35.364944 8984 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260926 01:55:35.366576 8981 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:35.368224 8984 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:35.371764 8985 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:55:35.380400 8985 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:55:35.375331 8984 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 3 FOLLOWER]: Advancing to term 4
I20260926 01:55:35.400981 8985 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:55:35.401355 8985 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:55:35.407362 8985 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:55:35.410385 8984 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:35.420806 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
I20260926 01:55:35.421561 8550 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 3 FOLLOWER]: Advancing to term 4
I20260926 01:55:35.423873 8985 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Loaded TSK: 0
I20260926 01:55:35.426259 8985 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:55:35.430187 8550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 4.
I20260926 01:55:35.432736 8679 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849; no voters:
I20260926 01:55:35.433849 8981 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 4 FOLLOWER]: Leader election won for term 4
I20260926 01:55:35.435113 8981 raft_consensus.cc:697] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 4 LEADER]: Becoming Leader. State: Replica: f75ba46482de422ea23daf17aa100849, State: Running, Role: LEADER
I20260926 01:55:35.436056 8981 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:35.450091 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:55:35.455968 8984 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 4 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:55:35.456395 8986 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75ba46482de422ea23daf17aa100849. Latest consensus state: current_term: 4 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:35.457156 8986 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:35.475889 8988 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:55:35.494010 8988 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:55:35.506886 8988 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:55:35.507198 8988 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:55:35.516615 8988 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:55:35.525979 8988 catalog_manager.cc:6230] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Loaded TSK: 0
I20260926 01:55:35.527460 8988 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:55:35.846237 8705 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 4 LEADER]: Rejecting Update request from peer 9ef98e01cb9a44408ebf207163f208b5 for earlier term 3. Current term is 4. Ops: []
I20260926 01:55:35.848682 8982 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260926 01:55:35.859028 8982 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [LEADER]: Peer f75ba46482de422ea23daf17aa100849 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260926 01:55:35.859823 8990 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260926 01:55:35.860244 8990 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9ef98e01cb9a44408ebf207163f208b5, State: Running, Role: LEADER
I20260926 01:55:35.861416 8990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:35.862800 8990 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 3 FOLLOWER]: Advancing to term 4
I20260926 01:55:35.936086 8550 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Refusing update from remote peer f75ba46482de422ea23daf17aa100849: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260926 01:55:35.938410 8986 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260926 01:55:35.964624 8991 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75ba46482de422ea23daf17aa100849. Latest consensus state: current_term: 4 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:35.965574 8991 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:35.993543 8991 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:35.994352 8991 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:36.000424 8986 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:36.001304 8986 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:36.001539 8625 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Refusing update from remote peer f75ba46482de422ea23daf17aa100849: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260926 01:55:36.001893 8625 pending_rounds.cc:85] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Aborting all ops after (but not including) 5
I20260926 01:55:36.002197 8625 pending_rounds.cc:107] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260926 01:55:36.002558 8625 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260926 01:55:36.004706 8993 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260926 01:55:36.026110 8990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75ba46482de422ea23daf17aa100849. Latest consensus state: current_term: 4 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:36.027024 8990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:36.028489 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:36.029225 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:36.370218 8952 heartbeater.cc:499] Master 127.8.76.188:38143 was elected leader, sending a full tablet report...
W20260926 01:55:36.286980 8972 debug-util.cc:398] Leaking SignalData structure 0x7b0800037180 after lost signal to thread 8962
W20260926 01:55:36.391708 8972 kernel_stack_watchdog.cc:198] Thread 8962 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260926 01:55:36.392695 8975 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1502 milliseconds
W20260926 01:55:36.392670 8962 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.506s user 0.373s sys 0.740s
W20260926 01:55:36.390528 8976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:36.393682 8962 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.508s user 0.373s sys 0.741s
I20260926 01:55:36.394078 8962 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260926 01:55:34.888055 8974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:36.398420 8962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:55:36.401196 8962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:55:36.402767 8962 hybrid_clock.cc:648] HybridClock initialized: now 1790387736402729 us; error 57 us; skew 500 ppm
I20260926 01:55:36.403666 8962 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:55:36.411302 8962 webserver.cc:533] Webserver started at http://127.8.76.130:40869/ using document root <none> and password file <none>
I20260926 01:55:36.412385 8962 fs_manager.cc:362] Metadata directory not provided
I20260926 01:55:36.412614 8962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:55:36.413105 8962 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:55:36.418023 8962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data/instance:
uuid: "d20f26c9295641ecbf77d290ca206bb2"
format_stamp: "Formatted at 2026-09-26 01:55:36 on dist-test-slave-5pk1"
I20260926 01:55:36.419595 8962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/instance:
uuid: "d20f26c9295641ecbf77d290ca206bb2"
format_stamp: "Formatted at 2026-09-26 01:55:36 on dist-test-slave-5pk1"
I20260926 01:55:36.428184 8962 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.000s sys 0.008s
I20260926 01:55:36.434885 9002 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:36.436419 8962 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 01:55:36.436794 8962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal
uuid: "d20f26c9295641ecbf77d290ca206bb2"
format_stamp: "Formatted at 2026-09-26 01:55:36 on dist-test-slave-5pk1"
I20260926 01:55:36.437173 8962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:55:36.492709 8991 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:36.493253 8991 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:36.498858 8962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:55:36.499204 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:36.501188 8962 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:55:36.501765 8962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:55:36.504520 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:36.511566 8982 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:36.512185 8982 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:36.517771 8982 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:36.495970 8991 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:36.519618 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:36.517129 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:36.520663 8962 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260926 01:55:36.511137 8524 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba; no voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849
I20260926 01:55:36.522313 8991 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:55:36.522810 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:36.523648 8982 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:55:36.527549 8962 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:55:36.527952 8962 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 01:55:36.528318 8962 ts_tablet_manager.cc:616] Registered 0 tablets
I20260926 01:55:36.528573 8962 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:36.576686 9013 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "f75ba46482de422ea23daf17aa100849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:36.580379 9013 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:36.771912 8962 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.130:41395
I20260926 01:55:36.771993 9117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.130:41395 every 8 connection(s)
I20260926 01:55:36.774857 8962 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/data/info.pb
I20260926 01:55:36.785118 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 8962
I20260926 01:55:36.785667 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/instance
I20260926 01:55:36.805900 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.76.131:0
--local_ip_for_outbound_sockets=127.8.76.131
--webserver_interface=127.8.76.131
--webserver_port=0
--tserver_master_addrs=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--builtin_ntp_servers=127.8.76.148:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260926 01:55:36.837047 9118 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:55:36.837668 9118 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:36.840090 9118 heartbeater.cc:507] Master 127.8.76.188:38143 requested a full tablet report, sending...
I20260926 01:55:36.843430 8695 ts_manager.cc:194] Registered new tserver with Master: d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:36.846236 8695 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.76.130:47173
I20260926 01:55:36.850033 9120 heartbeater.cc:344] Connected to a master server at 127.8.76.190:44589
I20260926 01:55:36.850427 9120 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:36.851328 9120 heartbeater.cc:507] Master 127.8.76.190:44589 requested a full tablet report, sending...
I20260926 01:55:36.854626 8540 ts_manager.cc:194] Registered new tserver with Master: d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:36.857908 9119 heartbeater.cc:344] Connected to a master server at 127.8.76.189:34163
I20260926 01:55:36.858309 9119 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:36.859009 9119 heartbeater.cc:507] Master 127.8.76.189:34163 requested a full tablet report, sending...
I20260926 01:55:36.861972 8615 ts_manager.cc:194] Registered new tserver with Master: d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:37.067014 9129 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:37.067557 9129 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:37.069996 9129 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:37.071456 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:37.071707 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:55:37.073536 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:37.074366 9129 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:55:37.193776 9128 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:55:37.194391 9128 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:55:37.195030 9128 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:55:37.231012 9128 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:55:37.232057 9128 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.76.131
I20260926 01:55:37.273396 9128 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.76.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.76.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.76.131
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:55:37.275190 9128 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:55:37.277750 9128 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:55:37.298324 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:37.352584 9118 heartbeater.cc:499] Master 127.8.76.188:38143 was elected leader, sending a full tablet report...
I20260926 01:55:37.555379 9141 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:37.556051 9141 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:37.558951 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:37.562585 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:37.564620 8524 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba; no voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849
I20260926 01:55:37.570158 9141 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:37.571470 9141 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:55:38.509212 9145 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:38.510239 9145 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:38.515533 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:38.520927 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:38.523417 8524 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba; no voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849
I20260926 01:55:38.528992 9145 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:38.530143 9145 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:55:37.301599 9136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:38.700562 9134 debug-util.cc:398] Leaking SignalData structure 0x7b0800037180 after lost signal to thread 9128
W20260926 01:55:38.909948 9134 kernel_stack_watchdog.cc:198] Thread 9128 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260926 01:55:38.914191 9128 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.614s user 0.613s sys 0.993s
W20260926 01:55:38.914659 9137 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1614 milliseconds
W20260926 01:55:38.914688 9128 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.614s user 0.613s sys 0.993s
W20260926 01:55:38.916496 9138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:38.916503 9128 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:55:38.917831 9128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:55:38.920279 9128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:55:38.921643 9128 hybrid_clock.cc:648] HybridClock initialized: now 1790387738921597 us; error 49 us; skew 500 ppm
I20260926 01:55:38.922825 9128 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:55:38.929868 9128 webserver.cc:533] Webserver started at http://127.8.76.131:37567/ using document root <none> and password file <none>
I20260926 01:55:38.930954 9128 fs_manager.cc:362] Metadata directory not provided
I20260926 01:55:38.931172 9128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:55:38.931676 9128 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:55:38.936475 9128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data/instance:
uuid: "bc00feec3f564608851f0112e97ab532"
format_stamp: "Formatted at 2026-09-26 01:55:38 on dist-test-slave-5pk1"
I20260926 01:55:38.937726 9128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/instance:
uuid: "bc00feec3f564608851f0112e97ab532"
format_stamp: "Formatted at 2026-09-26 01:55:38 on dist-test-slave-5pk1"
I20260926 01:55:38.946201 9128 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260926 01:55:38.952402 9151 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:38.953708 9128 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260926 01:55:38.954087 9128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal
uuid: "bc00feec3f564608851f0112e97ab532"
format_stamp: "Formatted at 2026-09-26 01:55:38 on dist-test-slave-5pk1"
I20260926 01:55:38.954523 9128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:55:39.006048 9128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:55:39.007858 9128 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:55:39.008324 9128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:55:39.011387 9128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260926 01:55:39.015990 9128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:55:39.016191 9128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:39.016459 9128 ts_tablet_manager.cc:616] Registered 0 tablets
I20260926 01:55:39.016613 9128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:39.054606 9177 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:39.055094 9177 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:39.056941 9177 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:39.058048 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:39.058334 8705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:55:39.060184 8524 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba; no voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849
I20260926 01:55:39.060979 9177 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:55:39.191955 9128 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.131:43045
I20260926 01:55:39.192066 9264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.131:43045 every 8 connection(s)
I20260926 01:55:39.194916 9128 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/data/info.pb
I20260926 01:55:39.201707 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 9128
I20260926 01:55:39.202427 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/instance
I20260926 01:55:39.245729 9265 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:55:39.246243 9265 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:39.247686 9265 heartbeater.cc:507] Master 127.8.76.188:38143 requested a full tablet report, sending...
I20260926 01:55:39.248988 9267 heartbeater.cc:344] Connected to a master server at 127.8.76.190:44589
I20260926 01:55:39.249310 9267 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:39.250106 9267 heartbeater.cc:507] Master 127.8.76.190:44589 requested a full tablet report, sending...
I20260926 01:55:39.252040 8695 ts_manager.cc:194] Registered new tserver with Master: bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:39.253238 8540 ts_manager.cc:194] Registered new tserver with Master: bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:39.253883 8695 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.76.131:46649
I20260926 01:55:39.254437 9266 heartbeater.cc:344] Connected to a master server at 127.8.76.189:34163
I20260926 01:55:39.254788 9266 heartbeater.cc:461] Registering TS with master...
I20260926 01:55:39.255625 9266 heartbeater.cc:507] Master 127.8.76.189:34163 requested a full tablet report, sending...
I20260926 01:55:39.257928 8615 ts_manager.cc:194] Registered new tserver with Master: bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:39.276477 8498 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260926 01:55:39.348606 8695 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59798:
name: "default.Table_e8fc0c2764f3477c97065d16cab079b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:39.349920 8692 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59798:
name: "default.Table_7d9023159f7d435f9087220b3abe3d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:39.348477 8693 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59798:
name: "default.Table_989e457ed74a45c79e57e50449a3762e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:39.351838 8694 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59798:
name: "default.Table_ace7c3ed00904d9188f11ff3fa52337c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:39.353411 8694 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace7c3ed00904d9188f11ff3fa52337c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:39.353051 8693 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_989e457ed74a45c79e57e50449a3762e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:39.355614 8692 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9023159f7d435f9087220b3abe3d3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:39.356963 8695 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8fc0c2764f3477c97065d16cab079b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:39.454854 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 8652
W20260926 01:55:39.497864 8601 connection.cc:573] server connection from 127.0.0.1:37296 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:55:39.759676 9265 heartbeater.cc:499] Master 127.8.76.188:38143 was elected leader, sending a full tablet report...
W20260926 01:55:39.764402 9265 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:38143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:39.888026 9309 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:39.888525 9309 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:39.890234 9309 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:39.891391 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:55:39.893426 8599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260926 01:55:39.897514 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:39.897900 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:39.898535 9309 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:55:39.924119 9312 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f75ba46482de422ea23daf17aa100849)
I20260926 01:55:39.924576 9312 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:39.928040 9312 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:39.928460 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:55:39.929154 8625 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 4.
I20260926 01:55:39.930575 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:55:39.931862 9312 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260926 01:55:39.932271 9312 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Starting leader election (detected failure of leader f75ba46482de422ea23daf17aa100849)
W20260926 01:55:39.932703 8952 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:38143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:39.932760 9312 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 4 FOLLOWER]: Advancing to term 5
W20260926 01:55:39.936355 8524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260926 01:55:39.940101 9312 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
W20260926 01:55:39.941496 8524 leader_election.cc:336] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:39.942930 9312 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 election: Requested vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:55:39.943243 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:55:39.943840 8625 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 4 FOLLOWER]: Advancing to term 5
W20260926 01:55:39.949124 8524 leader_election.cc:336] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:39.951300 8625 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 5.
I20260926 01:55:39.952442 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters: f75ba46482de422ea23daf17aa100849
I20260926 01:55:39.953379 9312 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 5 FOLLOWER]: Leader election won for term 5
I20260926 01:55:39.954524 9312 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 5 LEADER]: Becoming Leader. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:55:39.955358 9312 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
W20260926 01:55:39.959118 9118 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:38143 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:39.960451 9316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 5 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:39.961309 9316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:55:39.962764 9318 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:55:39.968341 9318 catalog_manager.cc:692] Loaded metadata for table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]
I20260926 01:55:39.974573 9318 tablet_loader.cc:96] loaded metadata for tablet d4974aeb98134f8d86991c801d17b887 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d])
I20260926 01:55:39.975827 9318 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:55:39.978726 9318 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:55:39.978952 9318 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:55:39.982214 9318 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:55:39.985190 9318 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loaded TSK: 0
I20260926 01:55:39.986047 9318 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:55:40.134931 8577 master_runner.cc:187] Error getting master registration for 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111),
I20260926 01:55:40.135408 8577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:55:40.483615 8625 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260926 01:55:40.485186 9322 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 01:55:40.494328 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 5 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:40.495193 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:40.512418 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:40.513196 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:55:40.516603 9326 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:55:40.517340 9326 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
W20260926 01:55:40.537998 8524 consensus_peers.cc:597] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba -> Peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Couldn't send request to peer f75ba46482de422ea23daf17aa100849. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:40.611783 9053 tablet_service.cc:1511] Processing CreateTablet for tablet d4974aeb98134f8d86991c801d17b887 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:40.614260 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4974aeb98134f8d86991c801d17b887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:40.613929 8887 tablet_service.cc:1511] Processing CreateTablet for tablet d4974aeb98134f8d86991c801d17b887 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:40.616701 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4974aeb98134f8d86991c801d17b887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:40.629165 9200 tablet_service.cc:1511] Processing CreateTablet for tablet d4974aeb98134f8d86991c801d17b887 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:40.631680 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4974aeb98134f8d86991c801d17b887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:40.653808 9340 tablet_bootstrap.cc:492] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:40.655966 9339 tablet_bootstrap.cc:492] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:40.664738 9339 tablet_bootstrap.cc:654] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:40.665479 9340 tablet_bootstrap.cc:654] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:40.667476 9339 log.cc:826] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Log is configured to *not* fsync() on all Append() calls
I20260926 01:55:40.668414 9340 log.cc:826] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Log is configured to *not* fsync() on all Append() calls
I20260926 01:55:40.671169 9341 tablet_bootstrap.cc:492] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:40.675589 9340 tablet_bootstrap.cc:492] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:40.676169 9340 ts_tablet_manager.cc:1403] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.004s
I20260926 01:55:40.678407 9339 tablet_bootstrap.cc:492] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:40.679098 9339 ts_tablet_manager.cc:1403] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.008s
I20260926 01:55:40.680744 9341 tablet_bootstrap.cc:654] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:40.683419 9341 log.cc:826] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Log is configured to *not* fsync() on all Append() calls
I20260926 01:55:40.698962 9341 tablet_bootstrap.cc:492] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:40.699640 9341 ts_tablet_manager.cc:1403] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.009s
I20260926 01:55:40.706003 9340 raft_consensus.cc:359] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:40.707108 9340 raft_consensus.cc:385] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:40.707444 9340 raft_consensus.cc:740] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:40.708547 9340 consensus_queue.cc:260] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:40.711208 9339 raft_consensus.cc:359] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:40.712366 9339 raft_consensus.cc:385] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:40.712734 9339 raft_consensus.cc:740] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:40.714064 9339 consensus_queue.cc:260] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:40.723273 9120 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
I20260926 01:55:40.725093 9339 ts_tablet_manager.cc:1434] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.046s user 0.037s sys 0.004s
I20260926 01:55:40.727375 8954 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
I20260926 01:55:40.728689 9341 raft_consensus.cc:359] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:40.729686 9341 raft_consensus.cc:385] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:40.730007 9341 raft_consensus.cc:740] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:40.731287 9341 consensus_queue.cc:260] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:40.736603 9340 ts_tablet_manager.cc:1434] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.060s user 0.033s sys 0.015s
I20260926 01:55:40.741097 9341 ts_tablet_manager.cc:1434] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.041s user 0.036s sys 0.005s
W20260926 01:55:40.742954 9268 tablet.cc:2433] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:55:40.746097 9267 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
W20260926 01:55:40.801563 9125 tablet.cc:2433] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:55:40.812466 8957 tablet.cc:2433] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:55:39.491163 9275 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.188:38143)
W20260926 01:55:40.717648 8503 debug-util.cc:398] Leaking SignalData structure 0x7b080002d0a0 after lost signal to thread 9275
W20260926 01:55:40.825086 9275 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 1.33595e+06us
W20260926 01:55:40.826186 9275 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.188:38143)
I20260926 01:55:40.842231 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:38143
--webserver_interface=127.8.76.188
--webserver_port=33837
--builtin_ntp_servers=127.8.76.148:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260926 01:55:40.858599 9114 debug-util.cc:398] Leaking SignalData structure 0x7b08000adc80 after lost signal to thread 8967
W20260926 01:55:40.860908 9275 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.76.190:44589)
I20260926 01:55:40.864312 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_7d9023159f7d435f9087220b3abe3d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:40.864992 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_989e457ed74a45c79e57e50449a3762e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:40.865952 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_e8fc0c2764f3477c97065d16cab079b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:40.868417 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8fc0c2764f3477c97065d16cab079b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:40.868736 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_ace7c3ed00904d9188f11ff3fa52337c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:40.869175 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9023159f7d435f9087220b3abe3d3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:40.870571 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace7c3ed00904d9188f11ff3fa52337c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:40.868395 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_989e457ed74a45c79e57e50449a3762e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:40.880204 8536 master_service.cc:960] ReplaceTablet: received request to replace tablet d4974aeb98134f8d86991c801d17b887 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:40.882176 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet d4974aeb98134f8d86991c801d17b887 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:40.884934 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_fb5ef6736b214f64b5b9592a920d01e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:40.886837 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb5ef6736b214f64b5b9592a920d01e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:40.899905 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_7d9023159f7d435f9087220b3abe3d3a" } new_table_name: "default.Table_39851286a7944831b873cd10eb21d348" modify_external_catalogs: true
I20260926 01:55:40.925432 8536 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet d4974aeb98134f8d86991c801d17b887
I20260926 01:55:40.928489 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet d4974aeb98134f8d86991c801d17b887 with delete_type TABLET_DATA_DELETED (replaced by tablet 64fca98cdc94414c8a1356092da6e059) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:40.930011 8887 tablet_service.cc:1558] Processing DeleteTablet for tablet d4974aeb98134f8d86991c801d17b887 with delete_type TABLET_DATA_DELETED (replaced by tablet 64fca98cdc94414c8a1356092da6e059) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:40.931221 8536 catalog_manager.cc:6934] ReplaceTablet: tablet d4974aeb98134f8d86991c801d17b887 deleted and replaced by tablet 64fca98cdc94414c8a1356092da6e059
I20260926 01:55:40.931949 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet d4974aeb98134f8d86991c801d17b887 with delete_type TABLET_DATA_DELETED (replaced by tablet 64fca98cdc94414c8a1356092da6e059) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:40.934285 8535 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_ace7c3ed00904d9188f11ff3fa52337c" } new_table_name: "default.Table_52ecfcf7831d43518312fce28f2c2187" modify_external_catalogs: true
I20260926 01:55:40.946935 9352 tablet_replica.cc:333] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:40.947954 9352 raft_consensus.cc:2243] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:40.948598 9352 raft_consensus.cc:2272] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:40.958293 9352 ts_tablet_manager.cc:1916] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:40.960072 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_fb5ef6736b214f64b5b9592a920d01e2" } modify_external_catalogs: true
I20260926 01:55:40.960942 9353 tablet_replica.cc:333] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:40.960752 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_fb5ef6736b214f64b5b9592a920d01e2" } modify_external_catalogs: true
I20260926 01:55:40.961907 9353 raft_consensus.cc:2243] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:40.965387 9353 raft_consensus.cc:2272] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:40.967289 9351 tablet_replica.cc:333] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:40.968657 8540 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_989e457ed74a45c79e57e50449a3762e" } modify_external_catalogs: true
I20260926 01:55:40.969210 8540 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_989e457ed74a45c79e57e50449a3762e" } modify_external_catalogs: true
I20260926 01:55:40.971352 9351 raft_consensus.cc:2243] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:40.975922 9351 raft_consensus.cc:2272] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:40.976643 9353 ts_tablet_manager.cc:1916] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:40.982678 8539 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f736d44763904285822c959aceba858a has no consensus state
I20260926 01:55:40.995090 9351 ts_tablet_manager.cc:1916] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:40.997180 9352 ts_tablet_manager.cc:1929] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:40.997793 9352 log.cc:1199] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/d4974aeb98134f8d86991c801d17b887
I20260926 01:55:41.002844 8887 tablet_service.cc:1511] Processing CreateTablet for tablet f736d44763904285822c959aceba858a (DEFAULT_TABLE table=default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.004773 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f736d44763904285822c959aceba858a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.004705 9200 tablet_service.cc:1511] Processing CreateTablet for tablet f736d44763904285822c959aceba858a (DEFAULT_TABLE table=default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.005929 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f736d44763904285822c959aceba858a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.014118 9053 tablet_service.cc:1511] Processing CreateTablet for tablet f736d44763904285822c959aceba858a (DEFAULT_TABLE table=default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.015390 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f736d44763904285822c959aceba858a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.018529 9352 ts_tablet_manager.cc:1950] T d4974aeb98134f8d86991c801d17b887 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:41.022219 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet d4974aeb98134f8d86991c801d17b887 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]) successfully deleted
I20260926 01:55:41.025084 9340 tablet_bootstrap.cc:492] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:41.025995 8538 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 7c0e1e2372814bd3b9980bfd6fd25cc7
W20260926 01:55:41.027235 8535 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c24566a10bf3456bb398c0de0fea68b9 has no consensus state
W20260926 01:55:41.030982 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet f736d44763904285822c959aceba858a (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f736d44763904285822c959aceba858a consensus state has no leader
I20260926 01:55:41.031934 9340 tablet_bootstrap.cc:654] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.045349 9340 tablet_bootstrap.cc:492] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:41.045388 8540 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 0fa09efb8c234e96a5a7efc6543c9e12
I20260926 01:55:41.045941 9340 ts_tablet_manager.cc:1403] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20260926 01:55:41.048913 9340 raft_consensus.cc:359] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.049650 9340 raft_consensus.cc:385] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.049957 9340 raft_consensus.cc:740] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.050935 9340 consensus_queue.cc:260] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.053239 9340 ts_tablet_manager.cc:1434] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260926 01:55:41.057308 9341 tablet_bootstrap.cc:492] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:41.066498 9341 tablet_bootstrap.cc:654] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.071178 9351 ts_tablet_manager.cc:1929] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.071663 9351 log.cc:1199] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/d4974aeb98134f8d86991c801d17b887
I20260926 01:55:41.072590 9351 ts_tablet_manager.cc:1950] T d4974aeb98134f8d86991c801d17b887 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:41.076586 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet d4974aeb98134f8d86991c801d17b887 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]) successfully deleted
I20260926 01:55:41.077286 8540 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet f736d44763904285822c959aceba858a (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]). Expected version 1 got 0
I20260926 01:55:41.080564 8887 tablet_service.cc:1511] Processing CreateTablet for tablet c24566a10bf3456bb398c0de0fea68b9 (DEFAULT_TABLE table=default.Table_52ecfcf7831d43518312fce28f2c2187 [id=ecc5d3a25ca44ab99f692c841f786b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.081782 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24566a10bf3456bb398c0de0fea68b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.081899 9200 tablet_service.cc:1511] Processing CreateTablet for tablet c24566a10bf3456bb398c0de0fea68b9 (DEFAULT_TABLE table=default.Table_52ecfcf7831d43518312fce28f2c2187 [id=ecc5d3a25ca44ab99f692c841f786b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.083014 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24566a10bf3456bb398c0de0fea68b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.087589 8886 tablet_service.cc:1511] Processing CreateTablet for tablet 64fca98cdc94414c8a1356092da6e059 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.088915 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fca98cdc94414c8a1356092da6e059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.090890 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 64fca98cdc94414c8a1356092da6e059 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.092187 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fca98cdc94414c8a1356092da6e059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.084857 9339 tablet_bootstrap.cc:492] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:41.099622 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_9e1d94a0f56b45e7ba3ff8983e206a4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:41.103902 8535 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet f736d44763904285822c959aceba858a (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]). Expected version 1 got 0
I20260926 01:55:41.104435 9340 tablet_bootstrap.cc:492] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:55:41.105926 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e1d94a0f56b45e7ba3ff8983e206a4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.105937 9052 tablet_service.cc:1511] Processing CreateTablet for tablet c24566a10bf3456bb398c0de0fea68b9 (DEFAULT_TABLE table=default.Table_52ecfcf7831d43518312fce28f2c2187 [id=ecc5d3a25ca44ab99f692c841f786b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.107982 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24566a10bf3456bb398c0de0fea68b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.111050 9340 tablet_bootstrap.cc:654] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.112252 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 64fca98cdc94414c8a1356092da6e059 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.113510 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fca98cdc94414c8a1356092da6e059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.120375 9340 tablet_bootstrap.cc:492] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:41.120904 9340 ts_tablet_manager.cc:1403] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.012s
I20260926 01:55:41.124010 9340 raft_consensus.cc:359] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.124753 9340 raft_consensus.cc:385] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.125058 9340 raft_consensus.cc:740] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.126080 9340 consensus_queue.cc:260] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.129025 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet c24566a10bf3456bb398c0de0fea68b9 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:41.129206 9339 tablet_bootstrap.cc:654] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.130664 9340 ts_tablet_manager.cc:1434] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260926 01:55:41.110316 9353 ts_tablet_manager.cc:1929] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.131744 9353 log.cc:1199] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/d4974aeb98134f8d86991c801d17b887
I20260926 01:55:41.133000 9353 ts_tablet_manager.cc:1950] T d4974aeb98134f8d86991c801d17b887 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:41.138301 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_36f849eee0734c3fa7c7a8afb540d5e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:41.140208 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36f849eee0734c3fa7c7a8afb540d5e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.140295 9345 raft_consensus.cc:493] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:41.140575 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet d4974aeb98134f8d86991c801d17b887 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]) successfully deleted
I20260926 01:55:41.140944 9345 raft_consensus.cc:515] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.149887 9345 leader_election.cc:290] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:41.155288 9345 raft_consensus.cc:493] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:41.156201 9345 raft_consensus.cc:515] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.158370 9345 leader_election.cc:290] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:41.159785 8539 master_service.cc:960] ReplaceTablet: received request to replace tablet f736d44763904285822c959aceba858a from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:41.164202 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8992c2f5dc79410388cfe23af8c34f8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:41.174060 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_65ffc79c04794db78e6df3d433c9f290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:41.175571 9341 tablet_bootstrap.cc:492] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:41.176074 9341 ts_tablet_manager.cc:1403] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.119s user 0.019s sys 0.001s
W20260926 01:55:41.177112 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65ffc79c04794db78e6df3d433c9f290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:41.166157 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8992c2f5dc79410388cfe23af8c34f8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.188364 9341 raft_consensus.cc:359] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.190104 8534 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_9e1d94a0f56b45e7ba3ff8983e206a4b" } new_table_name: "default.Table_1230f111a018421b8e7f8ed355349412" modify_external_catalogs: true
I20260926 01:55:41.189138 9341 raft_consensus.cc:385] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.193208 9341 raft_consensus.cc:740] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.193907 9341 consensus_queue.cc:260] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.197769 9341 ts_tablet_manager.cc:1434] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.021s user 0.004s sys 0.005s
I20260926 01:55:41.198964 9341 tablet_bootstrap.cc:492] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:41.202337 8535 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet c24566a10bf3456bb398c0de0fea68b9
I20260926 01:55:41.204288 8535 catalog_manager.cc:6934] ReplaceTablet: tablet c24566a10bf3456bb398c0de0fea68b9 deleted and replaced by tablet 33a8c42ae9cd47baa5a4e437f5a37120
I20260926 01:55:41.207149 9341 tablet_bootstrap.cc:654] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.226269 9340 tablet_bootstrap.cc:492] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:41.229959 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_39851286a7944831b873cd10eb21d348" } modify_external_catalogs: true
I20260926 01:55:41.230573 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_39851286a7944831b873cd10eb21d348" } modify_external_catalogs: true
I20260926 01:55:41.231564 8887 tablet_service.cc:1558] Processing DeleteTablet for tablet c24566a10bf3456bb398c0de0fea68b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8c42ae9cd47baa5a4e437f5a37120) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:41.232715 9352 tablet_replica.cc:333] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:41.232860 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_36f849eee0734c3fa7c7a8afb540d5e0" } new_table_name: "default.Table_2c2ba4c9f9624868b50fa92b262ac749" modify_external_catalogs: true
I20260926 01:55:41.233588 9352 raft_consensus.cc:2243] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.234251 9352 raft_consensus.cc:2272] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.236949 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet c24566a10bf3456bb398c0de0fea68b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8c42ae9cd47baa5a4e437f5a37120) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:41.239576 9352 ts_tablet_manager.cc:1916] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:41.239626 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet c24566a10bf3456bb398c0de0fea68b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c24566a10bf3456bb398c0de0fea68b9 already in progress: creating tablet
I20260926 01:55:41.250217 9340 tablet_bootstrap.cc:654] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.249984 8540 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_52ecfcf7831d43518312fce28f2c2187" } modify_external_catalogs: true
I20260926 01:55:41.250625 8540 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_52ecfcf7831d43518312fce28f2c2187" } modify_external_catalogs: true
I20260926 01:55:41.281255 9352 ts_tablet_manager.cc:1929] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.281800 9352 log.cc:1199] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/c24566a10bf3456bb398c0de0fea68b9
I20260926 01:55:41.282790 9352 ts_tablet_manager.cc:1950] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:41.282817 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c24566a10bf3456bb398c0de0fea68b9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:41.284891 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f736d44763904285822c959aceba858a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:41.285769 9219 raft_consensus.cc:2468] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:41.288149 8539 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet f736d44763904285822c959aceba858a
W20260926 01:55:41.288127 8843 leader_election.cc:343] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:41.289068 8843 leader_election.cc:304] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:41.289768 8539 catalog_manager.cc:6934] ReplaceTablet: tablet f736d44763904285822c959aceba858a deleted and replaced by tablet f7dcb2358e434f36a831352105863807
I20260926 01:55:41.290233 9345 raft_consensus.cc:2804] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:41.292809 9345 raft_consensus.cc:493] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:41.293174 9345 raft_consensus.cc:3060] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:41.294953 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet f736d44763904285822c959aceba858a with delete_type TABLET_DATA_DELETED (replaced by tablet f7dcb2358e434f36a831352105863807) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:41.303897 9352 tablet_replica.cc:333] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:41.304370 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet f736d44763904285822c959aceba858a with delete_type TABLET_DATA_DELETED (replaced by tablet f7dcb2358e434f36a831352105863807) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:41.305292 9351 tablet_replica.cc:333] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:41.305157 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet c24566a10bf3456bb398c0de0fea68b9 (table default.Table_52ecfcf7831d43518312fce28f2c2187 [id=ecc5d3a25ca44ab99f692c841f786b44]) successfully deleted
I20260926 01:55:41.306002 9351 raft_consensus.cc:2243] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.306542 9351 raft_consensus.cc:2272] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.311004 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet f736d44763904285822c959aceba858a with delete_type TABLET_DATA_DELETED (replaced by tablet f7dcb2358e434f36a831352105863807) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:41.311869 9351 ts_tablet_manager.cc:1916] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:41.313267 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet f736d44763904285822c959aceba858a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f736d44763904285822c959aceba858a already in progress: creating tablet
I20260926 01:55:41.317595 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet c24566a10bf3456bb398c0de0fea68b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8c42ae9cd47baa5a4e437f5a37120) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:41.319803 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet c24566a10bf3456bb398c0de0fea68b9 because the tablet was not found. No further retry: Not found: Tablet not found: c24566a10bf3456bb398c0de0fea68b9
I20260926 01:55:41.325987 9345 raft_consensus.cc:515] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.328399 9345 leader_election.cc:290] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:41.330814 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f736d44763904285822c959aceba858a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:55:41.332106 8534 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbe24f2921db43ff987ca93cbdad2f42 has no consensus state
W20260926 01:55:41.332373 8843 leader_election.cc:343] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: Tablet not RUNNING: STOPPED
W20260926 01:55:41.333528 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10fae6a23dbd4ce4adb012817384f2e7 has no consensus state
I20260926 01:55:41.333598 9351 ts_tablet_manager.cc:1929] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.333962 9351 log.cc:1199] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/f736d44763904285822c959aceba858a
I20260926 01:55:41.338660 9352 raft_consensus.cc:2243] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.341781 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f736d44763904285822c959aceba858a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:41.350668 9351 ts_tablet_manager.cc:1950] T f736d44763904285822c959aceba858a P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:41.350956 9339 tablet_bootstrap.cc:492] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:41.351394 9339 ts_tablet_manager.cc:1403] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.267s user 0.006s sys 0.010s
I20260926 01:55:41.353936 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet f736d44763904285822c959aceba858a (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]) successfully deleted
I20260926 01:55:41.354903 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f736d44763904285822c959aceba858a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:41.355028 9339 raft_consensus.cc:359] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.355649 9339 raft_consensus.cc:385] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.355940 9339 raft_consensus.cc:740] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.357342 9339 consensus_queue.cc:260] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.358219 9073 raft_consensus.cc:3060] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:41.358426 8843 leader_election.cc:343] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: c24566a10bf3456bb398c0de0fea68b9
I20260926 01:55:41.358917 8843 leader_election.cc:304] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:41.359462 9352 raft_consensus.cc:2272] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.367933 8540 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 33a8c42ae9cd47baa5a4e437f5a37120
I20260926 01:55:41.367998 9051 tablet_service.cc:1511] Processing CreateTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 (DEFAULT_TABLE table=default.Table_1230f111a018421b8e7f8ed355349412 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.369268 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe24f2921db43ff987ca93cbdad2f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.377547 9199 tablet_service.cc:1511] Processing CreateTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 (DEFAULT_TABLE table=default.Table_1230f111a018421b8e7f8ed355349412 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.378841 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe24f2921db43ff987ca93cbdad2f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.387965 9073 raft_consensus.cc:2468] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
W20260926 01:55:41.359639 8843 raft_consensus.cc:2686] T c24566a10bf3456bb398c0de0fea68b9 P 49bd320bf73f4252804ec867ec077510: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260926 01:55:41.391319 8843 leader_election.cc:343] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:41.392961 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet c24566a10bf3456bb398c0de0fea68b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8c42ae9cd47baa5a4e437f5a37120) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:41.392899 8887 tablet_service.cc:1511] Processing CreateTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 (DEFAULT_TABLE table=default.Table_1230f111a018421b8e7f8ed355349412 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.394289 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe24f2921db43ff987ca93cbdad2f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.397502 8538 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet f7dcb2358e434f36a831352105863807
I20260926 01:55:41.398422 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_aedac2a106704b52b7914ffdc67ddfee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:41.401427 9340 tablet_bootstrap.cc:492] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:41.401906 9340 ts_tablet_manager.cc:1403] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.176s user 0.028s sys 0.019s
I20260926 01:55:41.405692 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet f736d44763904285822c959aceba858a with delete_type TABLET_DATA_DELETED (replaced by tablet f7dcb2358e434f36a831352105863807) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:41.405525 9340 raft_consensus.cc:359] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:41.407688 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet f736d44763904285822c959aceba858a because the tablet was not found. No further retry: Not found: Tablet not found: f736d44763904285822c959aceba858a
W20260926 01:55:41.415777 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aedac2a106704b52b7914ffdc67ddfee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.417697 8843 leader_election.cc:304] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
W20260926 01:55:41.419103 8843 raft_consensus.cc:2686] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 01:55:41.419967 8538 master_service.cc:960] ReplaceTablet: received request to replace tablet 64fca98cdc94414c8a1356092da6e059 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:41.421049 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_295af734b12542cdabfe8aed4a19f8ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:41.423110 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_295af734b12542cdabfe8aed4a19f8ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.430316 9339 ts_tablet_manager.cc:1434] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.079s user 0.010s sys 0.002s
I20260926 01:55:41.406224 9340 raft_consensus.cc:385] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.431295 9340 raft_consensus.cc:740] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.432116 9340 consensus_queue.cc:260] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.434324 9340 ts_tablet_manager.cc:1434] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260926 01:55:41.449994 9339 tablet_bootstrap.cc:492] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:41.454757 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet f736d44763904285822c959aceba858a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:41 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:41.455595 9353 tablet_replica.cc:333] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:41.456261 9353 raft_consensus.cc:2243] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.456746 9353 raft_consensus.cc:2272] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.457032 9352 ts_tablet_manager.cc:1916] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.459192 9353 ts_tablet_manager.cc:1916] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.461669 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_c2f17033c99e4c2d9fa8181e746f5359"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:41.463457 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2f17033c99e4c2d9fa8181e746f5359 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.473054 9339 tablet_bootstrap.cc:654] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.479039 9340 tablet_bootstrap.cc:492] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:41.485373 9341 tablet_bootstrap.cc:492] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:41.485877 9341 ts_tablet_manager.cc:1403] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.287s user 0.032s sys 0.025s
I20260926 01:55:41.486799 9340 tablet_bootstrap.cc:654] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.489256 9341 raft_consensus.cc:359] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.489955 9341 raft_consensus.cc:385] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.495855 9353 ts_tablet_manager.cc:1929] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.496315 9353 log.cc:1199] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/f736d44763904285822c959aceba858a
I20260926 01:55:41.506214 9340 tablet_bootstrap.cc:492] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:41.506332 8536 master_service.cc:960] ReplaceTablet: received request to replace tablet cbe24f2921db43ff987ca93cbdad2f42 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:41.506680 9340 ts_tablet_manager.cc:1403] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.004s
I20260926 01:55:41.497061 9353 ts_tablet_manager.cc:1950] T f736d44763904285822c959aceba858a P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:41.511341 9341 raft_consensus.cc:740] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.512933 9341 consensus_queue.cc:260] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.515372 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_806bcb7bcfe245ecad99da04fa3bd85a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:41.510169 9340 raft_consensus.cc:359] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.518589 9340 raft_consensus.cc:385] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 01:55:41.519409 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_806bcb7bcfe245ecad99da04fa3bd85a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.518806 9340 raft_consensus.cc:740] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.520975 9340 consensus_queue.cc:260] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.526203 9339 tablet_bootstrap.cc:492] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:41.526687 9339 ts_tablet_manager.cc:1403] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.077s user 0.013s sys 0.007s
I20260926 01:55:41.529737 9339 raft_consensus.cc:359] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.530395 9339 raft_consensus.cc:385] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.532430 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet f736d44763904285822c959aceba858a (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]) successfully deleted
I20260926 01:55:41.536021 9340 ts_tablet_manager.cc:1434] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260926 01:55:41.540580 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_2c2ba4c9f9624868b50fa92b262ac749" } new_table_name: "default.Table_d7233ccd12524b5eae620cd58c593335" modify_external_catalogs: true
I20260926 01:55:41.541193 8532 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_1230f111a018421b8e7f8ed355349412" } new_table_name: "default.Table_9d789f8478fc4820b5b2bbf2c72d3d00" modify_external_catalogs: true
I20260926 01:55:41.543211 9341 ts_tablet_manager.cc:1434] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.057s user 0.007s sys 0.004s
I20260926 01:55:41.530674 9339 raft_consensus.cc:740] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.549186 9339 consensus_queue.cc:260] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.556871 9339 ts_tablet_manager.cc:1434] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.030s user 0.002s sys 0.004s
I20260926 01:55:41.562507 9339 tablet_bootstrap.cc:492] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:41.563316 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 10fae6a23dbd4ce4adb012817384f2e7 (DEFAULT_TABLE table=default.Table_2c2ba4c9f9624868b50fa92b262ac749 [id=c8194b05bc344ec88191d0ae47dd3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.564579 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fae6a23dbd4ce4adb012817384f2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.566759 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet c24566a10bf3456bb398c0de0fea68b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8c42ae9cd47baa5a4e437f5a37120) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:41.567589 9351 tablet_replica.cc:333] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:41.558444 9341 tablet_bootstrap.cc:492] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:41.580600 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 10fae6a23dbd4ce4adb012817384f2e7 (DEFAULT_TABLE table=default.Table_2c2ba4c9f9624868b50fa92b262ac749 [id=c8194b05bc344ec88191d0ae47dd3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.582063 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fae6a23dbd4ce4adb012817384f2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.588109 8539 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_aedac2a106704b52b7914ffdc67ddfee" } modify_external_catalogs: true
I20260926 01:55:41.591537 9052 tablet_service.cc:1511] Processing CreateTablet for tablet d49f4812f2b3457890a51214acf2abab (DEFAULT_TABLE table=default.Table_65ffc79c04794db78e6df3d433c9f290 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.592707 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d49f4812f2b3457890a51214acf2abab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.592572 9200 tablet_service.cc:1511] Processing CreateTablet for tablet d49f4812f2b3457890a51214acf2abab (DEFAULT_TABLE table=default.Table_65ffc79c04794db78e6df3d433c9f290 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.593695 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d49f4812f2b3457890a51214acf2abab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.596877 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 7009b837b5ac466d837e1aacebe88713 (DEFAULT_TABLE table=default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.598136 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7009b837b5ac466d837e1aacebe88713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.599450 9339 tablet_bootstrap.cc:654] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.612052 8887 tablet_service.cc:1511] Processing CreateTablet for tablet 10fae6a23dbd4ce4adb012817384f2e7 (DEFAULT_TABLE table=default.Table_2c2ba4c9f9624868b50fa92b262ac749 [id=c8194b05bc344ec88191d0ae47dd3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.613333 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fae6a23dbd4ce4adb012817384f2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.614044 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 7009b837b5ac466d837e1aacebe88713 (DEFAULT_TABLE table=default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.615451 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7009b837b5ac466d837e1aacebe88713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.622572 9351 raft_consensus.cc:2243] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.623165 9351 raft_consensus.cc:2272] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.625679 9351 ts_tablet_manager.cc:1916] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.625877 8886 tablet_service.cc:1511] Processing CreateTablet for tablet d49f4812f2b3457890a51214acf2abab (DEFAULT_TABLE table=default.Table_65ffc79c04794db78e6df3d433c9f290 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.627058 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d49f4812f2b3457890a51214acf2abab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.628366 9050 tablet_service.cc:1511] Processing CreateTablet for tablet f7dcb2358e434f36a831352105863807 (DEFAULT_TABLE table=default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.629448 9339 tablet_bootstrap.cc:492] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:41.629892 9339 ts_tablet_manager.cc:1403] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.003s
I20260926 01:55:41.630184 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7dcb2358e434f36a831352105863807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.588713 8539 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_aedac2a106704b52b7914ffdc67ddfee" } modify_external_catalogs: true
I20260926 01:55:41.633576 9339 raft_consensus.cc:359] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.634963 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_295af734b12542cdabfe8aed4a19f8ea" } modify_external_catalogs: true
I20260926 01:55:41.635623 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_295af734b12542cdabfe8aed4a19f8ea" } modify_external_catalogs: true
I20260926 01:55:41.649247 8536 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet cbe24f2921db43ff987ca93cbdad2f42
I20260926 01:55:41.651024 8536 catalog_manager.cc:6934] ReplaceTablet: tablet cbe24f2921db43ff987ca93cbdad2f42 deleted and replaced by tablet 03b827807def43b39b91d61a6dfd9386
I20260926 01:55:41.653472 9341 tablet_bootstrap.cc:654] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.634328 9339 raft_consensus.cc:385] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.654763 9339 raft_consensus.cc:740] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.655028 9352 ts_tablet_manager.cc:1929] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.655443 9352 log.cc:1199] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/f736d44763904285822c959aceba858a
I20260926 01:55:41.655582 9339 consensus_queue.cc:260] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.656133 9352 ts_tablet_manager.cc:1950] T f736d44763904285822c959aceba858a P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:41.656629 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b827807def43b39b91d61a6dfd9386) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:41.658983 8885 tablet_service.cc:1511] Processing CreateTablet for tablet 7009b837b5ac466d837e1aacebe88713 (DEFAULT_TABLE table=default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.660368 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7009b837b5ac466d837e1aacebe88713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.660348 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b827807def43b39b91d61a6dfd9386) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:41.661384 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet cbe24f2921db43ff987ca93cbdad2f42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe24f2921db43ff987ca93cbdad2f42 already in progress: creating tablet
I20260926 01:55:41.662307 9345 raft_consensus.cc:493] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:41.662815 9345 raft_consensus.cc:515] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.664804 9345 leader_election.cc:290] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:41.666873 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbe24f2921db43ff987ca93cbdad2f42" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:41.667526 9073 raft_consensus.cc:2468] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:41.668921 8843 leader_election.cc:304] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:41.669735 9345 raft_consensus.cc:2804] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:41.670059 9345 raft_consensus.cc:493] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:41.670346 9345 raft_consensus.cc:3060] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:41.674640 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_bff67bcef4e04092813a1a4f6da66152"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:41.675611 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbe24f2921db43ff987ca93cbdad2f42" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:41.676359 8884 tablet_service.cc:1511] Processing CreateTablet for tablet f7dcb2358e434f36a831352105863807 (DEFAULT_TABLE table=default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:41.676999 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bff67bcef4e04092813a1a4f6da66152 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.677657 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7dcb2358e434f36a831352105863807. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:41.678835 8843 leader_election.cc:343] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:41.679486 9345 raft_consensus.cc:515] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.682447 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbe24f2921db43ff987ca93cbdad2f42" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:41.683861 9073 raft_consensus.cc:3060] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:41.696393 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbe24f2921db43ff987ca93cbdad2f42" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:55:41.698031 8843 leader_election.cc:343] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:41.699632 9339 ts_tablet_manager.cc:1434] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.069s user 0.001s sys 0.008s
I20260926 01:55:41.700588 9339 tablet_bootstrap.cc:492] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:41.703644 9073 raft_consensus.cc:2468] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:41.707113 9345 leader_election.cc:290] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:41.710853 9197 tablet_service.cc:1511] Processing CreateTablet for tablet f7dcb2358e434f36a831352105863807 (DEFAULT_TABLE table=default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:41.712137 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7dcb2358e434f36a831352105863807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:41.720424 9341 tablet_bootstrap.cc:492] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:41.720898 9341 ts_tablet_manager.cc:1403] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.163s user 0.007s sys 0.010s
I20260926 01:55:41.723752 8843 leader_election.cc:304] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
W20260926 01:55:41.728729 8532 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03b827807def43b39b91d61a6dfd9386 has no consensus state
I20260926 01:55:41.730733 9345 raft_consensus.cc:2804] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:41.732820 9341 raft_consensus.cc:359] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.733546 9341 raft_consensus.cc:385] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.734094 9341 raft_consensus.cc:740] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.738490 9351 ts_tablet_manager.cc:1929] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.738940 9351 log.cc:1199] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/c24566a10bf3456bb398c0de0fea68b9
I20260926 01:55:41.739770 9351 ts_tablet_manager.cc:1950] T c24566a10bf3456bb398c0de0fea68b9 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:41.742978 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet c24566a10bf3456bb398c0de0fea68b9 (table default.Table_52ecfcf7831d43518312fce28f2c2187 [id=ecc5d3a25ca44ab99f692c841f786b44]) successfully deleted
I20260926 01:55:41.743310 9339 tablet_bootstrap.cc:654] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:41.745707 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 54756.8us
I20260926 01:55:41.746047 9345 raft_consensus.cc:697] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
W20260926 01:55:41.749459 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet c24566a10bf3456bb398c0de0fea68b9 because the tablet was not found. No further retry: Not found: Tablet not found: c24566a10bf3456bb398c0de0fea68b9
I20260926 01:55:41.757174 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b827807def43b39b91d61a6dfd9386) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:41.758162 9352 tablet_replica.cc:333] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:41.747087 9345 consensus_queue.cc:237] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.759320 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_6bcb6c89d7ba499e90436ceab11c20b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:41.761315 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bcb6c89d7ba499e90436ceab11c20b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.761137 9341 consensus_queue.cc:260] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.765858 9341 ts_tablet_manager.cc:1434] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.045s user 0.010s sys 0.000s
I20260926 01:55:41.766948 9341 tablet_bootstrap.cc:492] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:41.771083 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10fae6a23dbd4ce4adb012817384f2e7 consensus state has no leader
I20260926 01:55:41.773677 8538 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 64fca98cdc94414c8a1356092da6e059
I20260926 01:55:41.783952 8538 catalog_manager.cc:6934] ReplaceTablet: tablet 64fca98cdc94414c8a1356092da6e059 deleted and replaced by tablet 4411120140f3410c9f4e33e26da20d98
I20260926 01:55:41.785207 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 64fca98cdc94414c8a1356092da6e059 with delete_type TABLET_DATA_DELETED (replaced by tablet 4411120140f3410c9f4e33e26da20d98) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:41.791106 9352 raft_consensus.cc:2243] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:41.792663 9352 pending_rounds.cc:70] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Trying to abort 1 pending ops.
I20260926 01:55:41.793015 9352 pending_rounds.cc:77] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428190278324224 op_type: NO_OP noop_request { }
I20260926 01:55:41.793579 9352 raft_consensus.cc:2889] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:41.793964 9352 raft_consensus.cc:2272] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.798944 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 64fca98cdc94414c8a1356092da6e059 with delete_type TABLET_DATA_DELETED (replaced by tablet 4411120140f3410c9f4e33e26da20d98) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:41.799815 9351 tablet_replica.cc:333] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:41.800369 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 64fca98cdc94414c8a1356092da6e059 with delete_type TABLET_DATA_DELETED (replaced by tablet 4411120140f3410c9f4e33e26da20d98) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:41.800520 9351 raft_consensus.cc:2243] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.801014 9351 raft_consensus.cc:2272] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.801164 9353 tablet_replica.cc:333] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
W20260926 01:55:41.801644 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet cbe24f2921db43ff987ca93cbdad2f42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe24f2921db43ff987ca93cbdad2f42 already in progress: creating tablet
I20260926 01:55:41.801882 9353 raft_consensus.cc:2243] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.802368 9353 raft_consensus.cc:2272] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.804522 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet 64fca98cdc94414c8a1356092da6e059 with delete_type TABLET_DATA_DELETED (replaced by tablet 4411120140f3410c9f4e33e26da20d98) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:41.804823 9351 ts_tablet_manager.cc:1916] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.807199 9353 ts_tablet_manager.cc:1916] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.809190 9340 tablet_bootstrap.cc:492] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:41.816982 9341 tablet_bootstrap.cc:654] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.818256 9340 tablet_bootstrap.cc:654] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:41.828907 9375 consensus_queue.cc:1175] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "49bd320bf73f4252804ec867ec077510" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260926 01:55:41.831892 9339 tablet_bootstrap.cc:492] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:41.831499 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_6a7da4b61e7c428a8763e54c7ee9c870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:41.833256 9339 ts_tablet_manager.cc:1403] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.133s user 0.012s sys 0.004s
W20260926 01:55:41.833417 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a7da4b61e7c428a8763e54c7ee9c870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.837683 9339 raft_consensus.cc:359] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.844609 9352 ts_tablet_manager.cc:1916] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.853137 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b827807def43b39b91d61a6dfd9386) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:41.860162 9340 tablet_bootstrap.cc:492] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:41.860642 9340 ts_tablet_manager.cc:1403] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.003s
I20260926 01:55:41.863828 8539 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet e115b029dbe14404abaeee05451f4fc2
I20260926 01:55:41.865864 9341 tablet_bootstrap.cc:492] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:41.866335 9341 ts_tablet_manager.cc:1403] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.003s
I20260926 01:55:41.871083 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_9dba2c1067df4cf397890976a54f09cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:41.873143 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dba2c1067df4cf397890976a54f09cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:41.838383 9339 raft_consensus.cc:385] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.874830 9339 raft_consensus.cc:740] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.875468 9353 ts_tablet_manager.cc:1929] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.875865 9353 log.cc:1199] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/64fca98cdc94414c8a1356092da6e059
I20260926 01:55:41.875691 9340 raft_consensus.cc:359] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.876552 9353 ts_tablet_manager.cc:1950] T 64fca98cdc94414c8a1356092da6e059 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:41.876407 9340 raft_consensus.cc:385] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.877569 9340 raft_consensus.cc:740] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.878427 9339 consensus_queue.cc:260] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:41.880970 9341 raft_consensus.cc:359] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.881639 9341 raft_consensus.cc:385] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:41.881891 9341 raft_consensus.cc:740] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:41.882681 8540 master_service.cc:960] ReplaceTablet: received request to replace tablet cbe24f2921db43ff987ca93cbdad2f42 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:41.889213 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 64fca98cdc94414c8a1356092da6e059 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]) successfully deleted
I20260926 01:55:41.889935 9351 ts_tablet_manager.cc:1929] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.890357 9351 log.cc:1199] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/64fca98cdc94414c8a1356092da6e059
I20260926 01:55:41.891100 9351 ts_tablet_manager.cc:1950] T 64fca98cdc94414c8a1356092da6e059 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:41.893815 9352 ts_tablet_manager.cc:1929] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:41.894260 9352 log.cc:1199] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/cbe24f2921db43ff987ca93cbdad2f42
I20260926 01:55:41.894850 9353 tablet_replica.cc:333] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
W20260926 01:55:41.895125 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 64fca98cdc94414c8a1356092da6e059 because the tablet was not found. No further retry: Not found: Tablet not found: 64fca98cdc94414c8a1356092da6e059
I20260926 01:55:41.896075 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 64fca98cdc94414c8a1356092da6e059 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]) successfully deleted
I20260926 01:55:41.896617 9340 consensus_queue.cc:260] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.901906 9341 consensus_queue.cc:260] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:41.903290 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet c24566a10bf3456bb398c0de0fea68b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 33a8c42ae9cd47baa5a4e437f5a37120) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:41.912882 9341 ts_tablet_manager.cc:1434] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260926 01:55:41.913508 9352 ts_tablet_manager.cc:1950] T cbe24f2921db43ff987ca93cbdad2f42 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:41.916713 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet cbe24f2921db43ff987ca93cbdad2f42 (table default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:55:41.917793 9339 ts_tablet_manager.cc:1434] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.084s user 0.008s sys 0.002s
I20260926 01:55:41.918304 9353 raft_consensus.cc:2243] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.918881 9353 raft_consensus.cc:2272] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.924372 9353 ts_tablet_manager.cc:1916] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.928795 9341 tablet_bootstrap.cc:492] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:41.931872 9339 tablet_bootstrap.cc:492] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:41.937702 9341 tablet_bootstrap.cc:654] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.906890 9340 ts_tablet_manager.cc:1434] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.046s user 0.007s sys 0.004s
I20260926 01:55:41.941347 9340 tablet_bootstrap.cc:492] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:41.941459 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet cbe24f2921db43ff987ca93cbdad2f42 with delete_type TABLET_DATA_DELETED (replaced by tablet 03b827807def43b39b91d61a6dfd9386) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:41.942219 9351 tablet_replica.cc:333] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:41.942934 9351 raft_consensus.cc:2243] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.943445 9351 raft_consensus.cc:2272] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.946054 9352 tablet_replica.cc:333] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:41.946010 9351 ts_tablet_manager.cc:1916] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.954972 9352 raft_consensus.cc:2243] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:41.955587 9352 raft_consensus.cc:2272] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:41.957674 9352 ts_tablet_manager.cc:1916] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:41.965533 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_65ffc79c04794db78e6df3d433c9f290" } new_table_name: "default.Table_fc05e029f86e4b1fa2b981970ef38da8" modify_external_catalogs: true
I20260926 01:55:41.969425 9339 tablet_bootstrap.cc:654] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.974212 9351 ts_tablet_manager.cc:1929] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.974699 9351 log.cc:1199] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/cbe24f2921db43ff987ca93cbdad2f42
I20260926 01:55:41.975405 9351 ts_tablet_manager.cc:1950] T cbe24f2921db43ff987ca93cbdad2f42 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:41.976251 9340 tablet_bootstrap.cc:654] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:41.978446 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet cbe24f2921db43ff987ca93cbdad2f42 (table default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:55:41.981748 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_806bcb7bcfe245ecad99da04fa3bd85a" } new_table_name: "default.Table_d482559c4505477fa6df7f7baab2e03f" modify_external_catalogs: true
I20260926 01:55:41.982712 8535 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:41.984193 9353 ts_tablet_manager.cc:1929] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:41.988009 9353 log.cc:1199] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/cbe24f2921db43ff987ca93cbdad2f42
I20260926 01:55:41.988731 9353 ts_tablet_manager.cc:1950] T cbe24f2921db43ff987ca93cbdad2f42 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:41.993845 9341 tablet_bootstrap.cc:492] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:41.994359 9341 ts_tablet_manager.cc:1403] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.066s user 0.022s sys 0.024s
I20260926 01:55:41.995676 8533 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:42.009801 8534 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 0662497e0502488aac8ca9ebf7ffdc1b
I20260926 01:55:42.013561 9353 tablet_replica.cc:333] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:42.014266 9353 raft_consensus.cc:2243] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.014776 9353 raft_consensus.cc:2272] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.015518 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet cbe24f2921db43ff987ca93cbdad2f42 (table default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:55:41.998135 9341 raft_consensus.cc:359] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.024423 9341 raft_consensus.cc:385] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.024760 9341 raft_consensus.cc:740] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.025653 9341 consensus_queue.cc:260] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.027885 9340 tablet_bootstrap.cc:492] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:42.028327 9340 ts_tablet_manager.cc:1403] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.087s user 0.013s sys 0.002s
I20260926 01:55:42.039494 9341 ts_tablet_manager.cc:1434] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260926 01:55:42.043388 9340 raft_consensus.cc:359] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.044111 9340 raft_consensus.cc:385] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.044375 9340 raft_consensus.cc:740] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.046873 9341 tablet_bootstrap.cc:492] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:42.052121 8532 master_service.cc:960] ReplaceTablet: received request to replace tablet d49f4812f2b3457890a51214acf2abab from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:42.058488 9340 consensus_queue.cc:260] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.064016 9340 ts_tablet_manager.cc:1434] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s
I20260926 01:55:42.064965 9340 tablet_bootstrap.cc:492] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:55:42.068006 8527 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 03b827807def43b39b91d61a6dfd9386 has no consensus state
I20260926 01:55:42.072027 9353 ts_tablet_manager.cc:1916] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.074163 9339 tablet_bootstrap.cc:492] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:42.074636 9339 ts_tablet_manager.cc:1403] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.144s user 0.012s sys 0.012s
I20260926 01:55:42.078317 9339 raft_consensus.cc:359] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.079051 9339 raft_consensus.cc:385] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.079308 9339 raft_consensus.cc:740] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.079995 9339 consensus_queue.cc:260] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.087306 9352 ts_tablet_manager.cc:1929] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.091917 9353 ts_tablet_manager.cc:1929] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.092344 9353 log.cc:1199] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/c24566a10bf3456bb398c0de0fea68b9
I20260926 01:55:42.094426 9353 ts_tablet_manager.cc:1950] T c24566a10bf3456bb398c0de0fea68b9 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:42.094408 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8992c2f5dc79410388cfe23af8c34f8b" } modify_external_catalogs: true
I20260926 01:55:42.096194 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8992c2f5dc79410388cfe23af8c34f8b" } modify_external_catalogs: true
I20260926 01:55:42.097000 9341 tablet_bootstrap.cc:654] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.097612 8539 master_service.cc:960] ReplaceTablet: received request to replace tablet 7009b837b5ac466d837e1aacebe88713 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:42.087713 9352 log.cc:1199] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/64fca98cdc94414c8a1356092da6e059
I20260926 01:55:42.099120 9352 ts_tablet_manager.cc:1950] T 64fca98cdc94414c8a1356092da6e059 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:42.102792 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 64fca98cdc94414c8a1356092da6e059 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]) successfully deleted
I20260926 01:55:42.105924 9340 tablet_bootstrap.cc:654] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:42.108515 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d092195015a45c49cdcb60e343ff787 consensus state has no leader
I20260926 01:55:42.111079 9200 tablet_service.cc:1511] Processing CreateTablet for tablet dda6b1627dd846f08d717f154c3e23d9 (DEFAULT_TABLE table=default.Table_c2f17033c99e4c2d9fa8181e746f5359 [id=4560c208b2344ba5913dbd1d30a6d09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.112304 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda6b1627dd846f08d717f154c3e23d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.112272 8884 tablet_service.cc:1511] Processing CreateTablet for tablet dda6b1627dd846f08d717f154c3e23d9 (DEFAULT_TABLE table=default.Table_c2f17033c99e4c2d9fa8181e746f5359 [id=4560c208b2344ba5913dbd1d30a6d09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.113421 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda6b1627dd846f08d717f154c3e23d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.118749 9339 ts_tablet_manager.cc:1434] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260926 01:55:42.119555 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 4d092195015a45c49cdcb60e343ff787 (DEFAULT_TABLE table=default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.120661 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d092195015a45c49cdcb60e343ff787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.120790 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 03b827807def43b39b91d61a6dfd9386 (DEFAULT_TABLE table=default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.121805 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b827807def43b39b91d61a6dfd9386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.124435 9339 tablet_bootstrap.cc:492] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:42.125584 9051 tablet_service.cc:1511] Processing CreateTablet for tablet dda6b1627dd846f08d717f154c3e23d9 (DEFAULT_TABLE table=default.Table_c2f17033c99e4c2d9fa8181e746f5359 [id=4560c208b2344ba5913dbd1d30a6d09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.126749 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda6b1627dd846f08d717f154c3e23d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.130834 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 4d092195015a45c49cdcb60e343ff787 (DEFAULT_TABLE table=default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.135726 8885 tablet_service.cc:1511] Processing CreateTablet for tablet 4d092195015a45c49cdcb60e343ff787 (DEFAULT_TABLE table=default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.137398 8886 tablet_service.cc:1511] Processing CreateTablet for tablet 03b827807def43b39b91d61a6dfd9386 (DEFAULT_TABLE table=default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.139330 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet c24566a10bf3456bb398c0de0fea68b9 (table default.Table_52ecfcf7831d43518312fce28f2c2187 [id=ecc5d3a25ca44ab99f692c841f786b44]) successfully deleted
I20260926 01:55:42.140401 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 03b827807def43b39b91d61a6dfd9386 (DEFAULT_TABLE table=default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.144766 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d092195015a45c49cdcb60e343ff787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.148478 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b827807def43b39b91d61a6dfd9386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.151306 9339 tablet_bootstrap.cc:654] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.156060 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_e13d9ec3694c418e906e55984e87a263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:42.158509 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e13d9ec3694c418e906e55984e87a263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:42.162613 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d092195015a45c49cdcb60e343ff787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.165170 9341 tablet_bootstrap.cc:492] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:42.165637 9341 ts_tablet_manager.cc:1403] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.010s
I20260926 01:55:42.165702 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b827807def43b39b91d61a6dfd9386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.166879 9345 raft_consensus.cc:493] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:42.167318 9345 raft_consensus.cc:515] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.169327 9345 leader_election.cc:290] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:42.169129 9341 raft_consensus.cc:359] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.169795 9341 raft_consensus.cc:385] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.170068 9341 raft_consensus.cc:740] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.170742 9341 consensus_queue.cc:260] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.172361 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fae6a23dbd4ce4adb012817384f2e7" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:42.173709 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fae6a23dbd4ce4adb012817384f2e7" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:42.174326 9219 raft_consensus.cc:2468] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:42.176507 8843 leader_election.cc:304] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
W20260926 01:55:42.177539 8843 leader_election.cc:343] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:42.194486 9341 ts_tablet_manager.cc:1434] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260926 01:55:42.198704 9345 raft_consensus.cc:2804] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:42.199128 9345 raft_consensus.cc:493] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:42.199445 9345 raft_consensus.cc:3060] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:42.201072 9341 tablet_bootstrap.cc:492] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:42.217069 9341 tablet_bootstrap.cc:654] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.220459 9340 tablet_bootstrap.cc:492] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:42.220933 9340 ts_tablet_manager.cc:1403] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.156s user 0.021s sys 0.002s
I20260926 01:55:42.223507 9340 raft_consensus.cc:359] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.224179 9340 raft_consensus.cc:385] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.224434 9340 raft_consensus.cc:740] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.230974 9340 consensus_queue.cc:260] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.236306 8533 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_9dba2c1067df4cf397890976a54f09cd" } modify_external_catalogs: true
I20260926 01:55:42.236905 8533 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_9dba2c1067df4cf397890976a54f09cd" } modify_external_catalogs: true
I20260926 01:55:42.237860 9340 ts_tablet_manager.cc:1434] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260926 01:55:42.238950 9340 tablet_bootstrap.cc:492] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:42.241680 8887 tablet_service.cc:1511] Processing CreateTablet for tablet 450bace8066646f3833bcb4149b625b6 (DEFAULT_TABLE table=default.Table_9dba2c1067df4cf397890976a54f09cd [id=fae5d0ce53564f6ca4b731ecbb660a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.242902 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 450bace8066646f3833bcb4149b625b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.243391 9200 tablet_service.cc:1511] Processing CreateTablet for tablet 450bace8066646f3833bcb4149b625b6 (DEFAULT_TABLE table=default.Table_9dba2c1067df4cf397890976a54f09cd [id=fae5d0ce53564f6ca4b731ecbb660a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.244563 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 450bace8066646f3833bcb4149b625b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.243983 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_d25a39f7e9fa4682af9a020d33a580ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:42.246897 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d25a39f7e9fa4682af9a020d33a580ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:42.251433 9341 tablet_bootstrap.cc:492] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:42.251881 9341 ts_tablet_manager.cc:1403] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.000s
I20260926 01:55:42.258600 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 450bace8066646f3833bcb4149b625b6 (DEFAULT_TABLE table=default.Table_9dba2c1067df4cf397890976a54f09cd [id=fae5d0ce53564f6ca4b731ecbb660a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.259967 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 450bace8066646f3833bcb4149b625b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.262531 9374 raft_consensus.cc:493] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:42.262389 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 9bee4f08f4e6472b9ec711b23c67f7de (DEFAULT_TABLE table=default.Table_6a7da4b61e7c428a8763e54c7ee9c870 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.263041 9374 raft_consensus.cc:515] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.263291 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 9bee4f08f4e6472b9ec711b23c67f7de (DEFAULT_TABLE table=default.Table_6a7da4b61e7c428a8763e54c7ee9c870 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.263760 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bee4f08f4e6472b9ec711b23c67f7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.263581 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 9bee4f08f4e6472b9ec711b23c67f7de (DEFAULT_TABLE table=default.Table_6a7da4b61e7c428a8763e54c7ee9c870 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.264498 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bee4f08f4e6472b9ec711b23c67f7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.265630 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bee4f08f4e6472b9ec711b23c67f7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.265566 9374 leader_election.cc:290] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:42.268312 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49f4812f2b3457890a51214acf2abab" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:42.268997 9073 raft_consensus.cc:2468] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:42.269327 9341 raft_consensus.cc:359] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.270008 9341 raft_consensus.cc:385] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.270205 9341 raft_consensus.cc:740] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.272323 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49f4812f2b3457890a51214acf2abab" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:42.272926 9219 raft_consensus.cc:2468] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:42.274971 8843 leader_election.cc:304] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:42.276453 9374 raft_consensus.cc:2804] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:42.276764 9374 raft_consensus.cc:493] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:42.277050 9374 raft_consensus.cc:3060] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:42.284576 9374 raft_consensus.cc:515] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.289944 9374 leader_election.cc:290] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:42.291327 9341 consensus_queue.cc:260] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.292026 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49f4812f2b3457890a51214acf2abab" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:42.292608 9219 raft_consensus.cc:3060] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:42.277038 9340 tablet_bootstrap.cc:654] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.294090 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 051b59f528ea4879ad526476c45ffc6e (DEFAULT_TABLE table=default.Table_6bcb6c89d7ba499e90436ceab11c20b9 [id=def17c340f58498cb4282a7b8599d730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.295301 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 051b59f528ea4879ad526476c45ffc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.299522 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 051b59f528ea4879ad526476c45ffc6e (DEFAULT_TABLE table=default.Table_6bcb6c89d7ba499e90436ceab11c20b9 [id=def17c340f58498cb4282a7b8599d730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.300724 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 051b59f528ea4879ad526476c45ffc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.305022 9048 tablet_service.cc:1511] Processing CreateTablet for tablet 051b59f528ea4879ad526476c45ffc6e (DEFAULT_TABLE table=default.Table_6bcb6c89d7ba499e90436ceab11c20b9 [id=def17c340f58498cb4282a7b8599d730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.306322 9048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 051b59f528ea4879ad526476c45ffc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.306805 9341 ts_tablet_manager.cc:1434] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20260926 01:55:42.306865 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d49f4812f2b3457890a51214acf2abab" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:42.307431 9073 raft_consensus.cc:3060] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:42.315791 9341 tablet_bootstrap.cc:492] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:42.324417 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a (DEFAULT_TABLE table=default.Table_bff67bcef4e04092813a1a4f6da66152 [id=008c87e2e12f48519efdacc69466442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.325618 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1618eb529bcc4947b4c0e1e5504af71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.329222 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a (DEFAULT_TABLE table=default.Table_bff67bcef4e04092813a1a4f6da66152 [id=008c87e2e12f48519efdacc69466442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.331079 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1618eb529bcc4947b4c0e1e5504af71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.344565 9341 tablet_bootstrap.cc:654] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.347615 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet d49f4812f2b3457890a51214acf2abab
I20260926 01:55:42.347735 8539 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 7009b837b5ac466d837e1aacebe88713
I20260926 01:55:42.349328 8539 catalog_manager.cc:6934] ReplaceTablet: tablet 7009b837b5ac466d837e1aacebe88713 deleted and replaced by tablet e8f5c3bdad494f34a3b72ab693599c55
I20260926 01:55:42.351290 8532 catalog_manager.cc:6934] ReplaceTablet: tablet d49f4812f2b3457890a51214acf2abab deleted and replaced by tablet 8a011a9edfe94e04afb6ea0367dd979f
I20260926 01:55:42.352362 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_a7300910c319422e8977d7025c913c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:42.355180 8536 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a011a9edfe94e04afb6ea0367dd979f has no consensus state
I20260926 01:55:42.361583 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet d49f4812f2b3457890a51214acf2abab with delete_type TABLET_DATA_DELETED (replaced by tablet 8a011a9edfe94e04afb6ea0367dd979f) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:42.362550 9352 tablet_replica.cc:333] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:42.363297 9352 raft_consensus.cc:2243] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.363803 9352 raft_consensus.cc:2272] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.369784 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 7009b837b5ac466d837e1aacebe88713 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f5c3bdad494f34a3b72ab693599c55) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:42.370553 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet d49f4812f2b3457890a51214acf2abab with delete_type TABLET_DATA_DELETED (replaced by tablet 8a011a9edfe94e04afb6ea0367dd979f) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:42.371268 9351 tablet_replica.cc:333] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:42.371652 9341 tablet_bootstrap.cc:492] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:42.372078 9351 raft_consensus.cc:2243] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.372169 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_9d789f8478fc4820b5b2bbf2c72d3d00" } new_table_name: "default.Table_1c1a0d3391224ba291d16b4e18b5d1c0" modify_external_catalogs: true
I20260926 01:55:42.373536 9351 raft_consensus.cc:2272] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.374557 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet d49f4812f2b3457890a51214acf2abab with delete_type TABLET_DATA_DELETED (replaced by tablet 8a011a9edfe94e04afb6ea0367dd979f) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:42.376317 9351 ts_tablet_manager.cc:1916] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.377341 9353 tablet_replica.cc:333] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:42.380123 9352 ts_tablet_manager.cc:1916] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.382771 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 7009b837b5ac466d837e1aacebe88713 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f5c3bdad494f34a3b72ab693599c55) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:42.391541 8533 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 450bace8066646f3833bcb4149b625b6
I20260926 01:55:42.395318 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 450bace8066646f3833bcb4149b625b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:42.372081 9341 ts_tablet_manager.cc:1403] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.000s
I20260926 01:55:42.397867 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet 7009b837b5ac466d837e1aacebe88713 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f5c3bdad494f34a3b72ab693599c55) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:42.399887 9341 raft_consensus.cc:359] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.400559 9341 raft_consensus.cc:385] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.400805 9341 raft_consensus.cc:740] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.401497 9341 consensus_queue.cc:260] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.413439 9341 ts_tablet_manager.cc:1434] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260926 01:55:42.414340 9352 ts_tablet_manager.cc:1929] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.414975 9352 log.cc:1199] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/d49f4812f2b3457890a51214acf2abab
I20260926 01:55:42.415933 9352 ts_tablet_manager.cc:1950] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:42.423506 9341 tablet_bootstrap.cc:492] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:42.437058 9219 raft_consensus.cc:2468] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:42.438335 9073 raft_consensus.cc:2468] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:42.438899 9353 raft_consensus.cc:2243] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.440948 9353 raft_consensus.cc:2272] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.442605 9341 tablet_bootstrap.cc:654] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.443501 9353 ts_tablet_manager.cc:1916] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.452122 9351 ts_tablet_manager.cc:1929] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.452708 9351 log.cc:1199] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/7009b837b5ac466d837e1aacebe88713
I20260926 01:55:42.435992 9345 raft_consensus.cc:515] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.453075 8843 leader_election.cc:304] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:42.453387 9351 ts_tablet_manager.cc:1950] T 7009b837b5ac466d837e1aacebe88713 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:55:42.453758 8843 raft_consensus.cc:2686] T d49f4812f2b3457890a51214acf2abab P 49bd320bf73f4252804ec867ec077510: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 01:55:42.455049 9345 leader_election.cc:290] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:42.464563 9341 tablet_bootstrap.cc:492] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:42.465018 9341 ts_tablet_manager.cc:1403] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.007s
I20260926 01:55:42.465628 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fae6a23dbd4ce4adb012817384f2e7" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:55:42.467625 8843 leader_election.cc:343] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:42.468559 9351 tablet_replica.cc:333] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:42.469319 9351 raft_consensus.cc:2243] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.469776 9351 raft_consensus.cc:2272] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.467602 9341 raft_consensus.cc:359] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:42.470499 9341 raft_consensus.cc:385] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.470813 9341 raft_consensus.cc:740] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.471485 9341 consensus_queue.cc:260] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:55:42.474357 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7300910c319422e8977d7025c913c22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:42.475205 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 7009b837b5ac466d837e1aacebe88713 (table default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc]) successfully deleted
I20260926 01:55:42.476998 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8a5125daf77b4495b907f0f302cd559d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:42.478363 9341 ts_tablet_manager.cc:1434] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
W20260926 01:55:42.478969 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a5125daf77b4495b907f0f302cd559d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:42.480235 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 450bace8066646f3833bcb4149b625b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:42.482896 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fae6a23dbd4ce4adb012817384f2e7" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:42.483462 9219 raft_consensus.cc:3060] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:42.485510 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet d49f4812f2b3457890a51214acf2abab (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]) successfully deleted
W20260926 01:55:42.486229 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 450bace8066646f3833bcb4149b625b6 because the tablet was not found. No further retry: Not found: Tablet not found: 450bace8066646f3833bcb4149b625b6
W20260926 01:55:42.486907 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 7009b837b5ac466d837e1aacebe88713 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7009b837b5ac466d837e1aacebe88713 already in progress: creating tablet
I20260926 01:55:42.489959 9341 tablet_bootstrap.cc:492] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:42.491988 9351 ts_tablet_manager.cc:1916] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.493096 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 450bace8066646f3833bcb4149b625b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:42.514945 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a (DEFAULT_TABLE table=default.Table_bff67bcef4e04092813a1a4f6da66152 [id=008c87e2e12f48519efdacc69466442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.516270 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1618eb529bcc4947b4c0e1e5504af71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.520462 9341 tablet_bootstrap.cc:654] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.530905 9339 tablet_bootstrap.cc:492] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:42.531388 9339 ts_tablet_manager.cc:1403] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.407s user 0.044s sys 0.025s
I20260926 01:55:42.534354 9339 raft_consensus.cc:359] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.536140 9219 raft_consensus.cc:2468] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:42.539448 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 4411120140f3410c9f4e33e26da20d98 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.540640 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4411120140f3410c9f4e33e26da20d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.543205 8843 leader_election.cc:304] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:42.553702 9345 raft_consensus.cc:2804] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:42.554772 9345 raft_consensus.cc:697] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:42.555660 9345 consensus_queue.cc:237] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.535027 9339 raft_consensus.cc:385] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.562822 9339 raft_consensus.cc:740] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.565109 9341 tablet_bootstrap.cc:492] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:42.565582 9341 ts_tablet_manager.cc:1403] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.003s
I20260926 01:55:42.566150 9339 consensus_queue.cc:260] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.569118 9341 raft_consensus.cc:359] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.569783 9341 raft_consensus.cc:385] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.570062 9341 raft_consensus.cc:740] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.571601 8534 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet e8f5c3bdad494f34a3b72ab693599c55
I20260926 01:55:42.573024 9341 consensus_queue.cc:260] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.570093 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_bff67bcef4e04092813a1a4f6da66152" } new_table_name: "default.Table_e18c82d7ba2541ff8790bc644e02d7ac" modify_external_catalogs: true
I20260926 01:55:42.579294 9340 tablet_bootstrap.cc:492] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:42.580273 9340 ts_tablet_manager.cc:1403] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.342s user 0.020s sys 0.025s
I20260926 01:55:42.582119 8538 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:42.584782 9340 raft_consensus.cc:359] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.588956 9341 ts_tablet_manager.cc:1434] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.023s user 0.000s sys 0.007s
I20260926 01:55:42.591616 9351 ts_tablet_manager.cc:1929] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.592012 9351 log.cc:1199] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/d49f4812f2b3457890a51214acf2abab
I20260926 01:55:42.592680 9351 ts_tablet_manager.cc:1950] T d49f4812f2b3457890a51214acf2abab P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:42.595189 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet f7dcb2358e434f36a831352105863807 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:42.607288 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet d49f4812f2b3457890a51214acf2abab (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]) successfully deleted
I20260926 01:55:42.612046 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8a5125daf77b4495b907f0f302cd559d" } modify_external_catalogs: true
I20260926 01:55:42.612624 9341 tablet_bootstrap.cc:492] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:42.612905 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8a5125daf77b4495b907f0f302cd559d" } modify_external_catalogs: true
I20260926 01:55:42.619701 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet f7dcb2358e434f36a831352105863807 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:42.621685 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 450bace8066646f3833bcb4149b625b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 450bace8066646f3833bcb4149b625b6 already in progress: creating tablet
I20260926 01:55:42.623283 9351 tablet_replica.cc:333] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:42.624032 9351 raft_consensus.cc:2243] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.624481 9351 raft_consensus.cc:2272] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.603389 8887 tablet_service.cc:1558] Processing DeleteTablet for tablet d49f4812f2b3457890a51214acf2abab with delete_type TABLET_DATA_DELETED (replaced by tablet 8a011a9edfe94e04afb6ea0367dd979f) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:42.630365 9349 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:55:42.631017 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet d49f4812f2b3457890a51214acf2abab because the tablet was not found. No further retry: Not found: Tablet not found: d49f4812f2b3457890a51214acf2abab
W20260926 01:55:42.631356 9349 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260926 01:55:42.617240 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet f7dcb2358e434f36a831352105863807 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:42.632072 9349 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260926 01:55:42.632386 9352 tablet_replica.cc:333] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:42.633109 9352 raft_consensus.cc:2243] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.633575 9352 raft_consensus.cc:2272] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.636127 9352 ts_tablet_manager.cc:1916] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.638648 9351 ts_tablet_manager.cc:1916] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.616421 8885 tablet_service.cc:1511] Processing CreateTablet for tablet 4411120140f3410c9f4e33e26da20d98 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.643477 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4411120140f3410c9f4e33e26da20d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.643852 9341 tablet_bootstrap.cc:654] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.645938 9340 raft_consensus.cc:385] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.646283 9340 raft_consensus.cc:740] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.647161 9340 consensus_queue.cc:260] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.652396 9340 ts_tablet_manager.cc:1434] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.072s user 0.006s sys 0.001s
W20260926 01:55:42.659857 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1618eb529bcc4947b4c0e1e5504af71a consensus state has no leader
I20260926 01:55:42.665803 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 4411120140f3410c9f4e33e26da20d98 (DEFAULT_TABLE table=default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.667346 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4411120140f3410c9f4e33e26da20d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.675381 9339 ts_tablet_manager.cc:1434] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.144s user 0.017s sys 0.006s
I20260926 01:55:42.677131 9339 tablet_bootstrap.cc:492] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:42.694429 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet 10fae6a23dbd4ce4adb012817384f2e7 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:42.705317 8532 master_service.cc:960] ReplaceTablet: received request to replace tablet dda6b1627dd846f08d717f154c3e23d9 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:42.706744 8538 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet a9edae8808f94707a56926e131512ef2
I20260926 01:55:42.707147 9339 tablet_bootstrap.cc:654] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.715533 9351 ts_tablet_manager.cc:1929] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.734716 9351 log.cc:1199] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/f7dcb2358e434f36a831352105863807
I20260926 01:55:42.735705 9351 ts_tablet_manager.cc:1950] T f7dcb2358e434f36a831352105863807 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:42.737910 9339 tablet_bootstrap.cc:492] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:42.738425 9339 ts_tablet_manager.cc:1403] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.000s
I20260926 01:55:42.738471 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_caaf07e0f7d24f7388b358362fd1668f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:42.740525 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_caaf07e0f7d24f7388b358362fd1668f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:42.743177 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet f7dcb2358e434f36a831352105863807 (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]) successfully deleted
I20260926 01:55:42.744309 9339 raft_consensus.cc:359] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.746589 9339 raft_consensus.cc:385] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.747025 9339 raft_consensus.cc:740] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.751678 9340 tablet_bootstrap.cc:492] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:42.752630 9339 consensus_queue.cc:260] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:42.767096 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 099af935249a4310b29852a258661794 (DEFAULT_TABLE table=default.Table_e13d9ec3694c418e906e55984e87a263 [id=0a1b8452d5494c0ea17e1fca2259e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.768355 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099af935249a4310b29852a258661794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.773833 9339 ts_tablet_manager.cc:1434] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.035s user 0.002s sys 0.004s
I20260926 01:55:42.778095 9339 tablet_bootstrap.cc:492] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:42.779248 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 099af935249a4310b29852a258661794 (DEFAULT_TABLE table=default.Table_e13d9ec3694c418e906e55984e87a263 [id=0a1b8452d5494c0ea17e1fca2259e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.780377 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099af935249a4310b29852a258661794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.788825 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 099af935249a4310b29852a258661794 (DEFAULT_TABLE table=default.Table_e13d9ec3694c418e906e55984e87a263 [id=0a1b8452d5494c0ea17e1fca2259e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.790518 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 (DEFAULT_TABLE table=default.Table_d25a39f7e9fa4682af9a020d33a580ad [id=67e50edec70245399c670c1c6aa26688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.790815 8535 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_9d789f8478fc4820b5b2bbf2c72d3d00 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 2 got 0
I20260926 01:55:42.791651 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d9cb4de16da44e1b90ac79e9e32cb88. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:42.792745 8540 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03b827807def43b39b91d61a6dfd9386 consensus state has no leader
I20260926 01:55:42.795856 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_37ba6de1661d4b1f95e93ea4b3348633"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:42.795928 8539 catalog_manager.cc:5838] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:42.797765 8539 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:42.799073 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_d541e514029640269428f8bf5156eb58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:42.797780 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37ba6de1661d4b1f95e93ea4b3348633 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:42.801855 8537 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_6bcb6c89d7ba499e90436ceab11c20b9" } modify_external_catalogs: true
I20260926 01:55:42.803346 8537 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_6bcb6c89d7ba499e90436ceab11c20b9" } modify_external_catalogs: true
W20260926 01:55:42.807542 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d541e514029640269428f8bf5156eb58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:42.798363 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 (DEFAULT_TABLE table=default.Table_d25a39f7e9fa4682af9a020d33a580ad [id=67e50edec70245399c670c1c6aa26688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.811394 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d9cb4de16da44e1b90ac79e9e32cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.815135 9339 tablet_bootstrap.cc:654] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:42.821620 9349 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:55:42.822050 9349 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
I20260926 01:55:42.823273 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_66f4ea34f6154f8e93c1d3f472e1bbbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:42.825161 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66f4ea34f6154f8e93c1d3f472e1bbbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:42.822435 9349 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:55:42.830859 9349 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:55:42.838304 9341 tablet_bootstrap.cc:492] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:42.838753 9341 ts_tablet_manager.cc:1403] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.226s user 0.053s sys 0.026s
I20260926 01:55:42.841238 9341 raft_consensus.cc:359] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:42.841946 9341 raft_consensus.cc:385] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.842223 9341 raft_consensus.cc:740] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.843011 9341 consensus_queue.cc:260] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:42.850975 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet dda6b1627dd846f08d717f154c3e23d9
I20260926 01:55:42.851154 9340 tablet_bootstrap.cc:654] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.854085 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet dda6b1627dd846f08d717f154c3e23d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1171186cc294417db439bedf0ca4a6a2) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:42.854842 9351 tablet_replica.cc:333] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:42.855535 9351 raft_consensus.cc:2243] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:42.855978 9351 raft_consensus.cc:2272] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:42.856631 8532 catalog_manager.cc:6934] ReplaceTablet: tablet dda6b1627dd846f08d717f154c3e23d9 deleted and replaced by tablet 1171186cc294417db439bedf0ca4a6a2
I20260926 01:55:42.856935 8887 tablet_service.cc:1558] Processing DeleteTablet for tablet dda6b1627dd846f08d717f154c3e23d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1171186cc294417db439bedf0ca4a6a2) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:42.858194 9351 ts_tablet_manager.cc:1916] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:42.862079 9347 raft_consensus.cc:493] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:42.862625 9347 raft_consensus.cc:515] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.864528 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 051b59f528ea4879ad526476c45ffc6e
I20260926 01:55:42.866415 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 051b59f528ea4879ad526476c45ffc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:42.866808 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet dda6b1627dd846f08d717f154c3e23d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1171186cc294417db439bedf0ca4a6a2) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:42.870149 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 051b59f528ea4879ad526476c45ffc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:42.870841 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099af935249a4310b29852a258661794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.873684 8887 tablet_service.cc:1558] Processing DeleteTablet for tablet 051b59f528ea4879ad526476c45ffc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:42.876863 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_a7300910c319422e8977d7025c913c22" } new_table_name: "default.Table_27cdeab15da2426d85cbbb43441e39d6" modify_external_catalogs: true
I20260926 01:55:42.884260 8536 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 4d092195015a45c49cdcb60e343ff787 (table default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]). Expected version 1 got 0
I20260926 01:55:42.884742 8887 tablet_service.cc:1558] Processing DeleteTablet for tablet 7009b837b5ac466d837e1aacebe88713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:42.885958 9347 leader_election.cc:290] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:42.891003 9341 ts_tablet_manager.cc:1434] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20260926 01:55:42.908627 9200 tablet_service.cc:1511] Processing CreateTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 (DEFAULT_TABLE table=default.Table_d25a39f7e9fa4682af9a020d33a580ad [id=67e50edec70245399c670c1c6aa26688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.909577 9340 tablet_bootstrap.cc:492] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:42.909803 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d9cb4de16da44e1b90ac79e9e32cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.910033 9340 ts_tablet_manager.cc:1403] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.160s user 0.006s sys 0.005s
I20260926 01:55:42.910673 8536 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_c2f17033c99e4c2d9fa8181e746f5359" } modify_external_catalogs: true
I20260926 01:55:42.911391 8536 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_c2f17033c99e4c2d9fa8181e746f5359" } modify_external_catalogs: true
I20260926 01:55:42.913234 9340 raft_consensus.cc:359] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:42.913846 9340 raft_consensus.cc:385] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:42.914085 9340 raft_consensus.cc:740] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:42.914949 9353 ts_tablet_manager.cc:1929] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.915405 9353 log.cc:1199] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/d49f4812f2b3457890a51214acf2abab
I20260926 01:55:42.916461 9353 ts_tablet_manager.cc:1950] T d49f4812f2b3457890a51214acf2abab P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:42.918177 9341 tablet_bootstrap.cc:492] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:42.918646 9340 consensus_queue.cc:260] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:42.927582 9340 ts_tablet_manager.cc:1434] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.017s user 0.006s sys 0.003s
I20260926 01:55:42.934947 9340 tablet_bootstrap.cc:492] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:42.936834 9352 ts_tablet_manager.cc:1929] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.937263 9352 log.cc:1199] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/f7dcb2358e434f36a831352105863807
I20260926 01:55:42.937970 9352 ts_tablet_manager.cc:1950] T f7dcb2358e434f36a831352105863807 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:42.940615 9352 tablet_replica.cc:333] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:42.940933 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet f7dcb2358e434f36a831352105863807 (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]) successfully deleted
I20260926 01:55:42.952188 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 4d092195015a45c49cdcb60e343ff787 (table default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]). Expected version 1 got 0
I20260926 01:55:42.964124 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.965372 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a011a9edfe94e04afb6ea0367dd979f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.968564 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.969786 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a011a9edfe94e04afb6ea0367dd979f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:42.975077 9367 raft_consensus.cc:493] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:42.975669 9367 raft_consensus.cc:515] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:42.984898 9341 tablet_bootstrap.cc:654] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.987663 9351 ts_tablet_manager.cc:1929] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:42.942404 9340 tablet_bootstrap.cc:654] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:42.983515 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:42.999302 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a011a9edfe94e04afb6ea0367dd979f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:43.003243 9367 leader_election.cc:290] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:43.004344 9352 raft_consensus.cc:2243] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.004848 9352 raft_consensus.cc:2272] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.007057 9352 ts_tablet_manager.cc:1916] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.010275 9351 log.cc:1199] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/dda6b1627dd846f08d717f154c3e23d9
I20260926 01:55:43.011252 9351 ts_tablet_manager.cc:1950] T dda6b1627dd846f08d717f154c3e23d9 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:43.012774 9340 tablet_bootstrap.cc:492] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:43.013203 9340 ts_tablet_manager.cc:1403] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.003s
I20260926 01:55:43.014626 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet dda6b1627dd846f08d717f154c3e23d9 (table default.Table_c2f17033c99e4c2d9fa8181e746f5359 [id=4560c208b2344ba5913dbd1d30a6d09e]) successfully deleted
W20260926 01:55:43.015419 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 051b59f528ea4879ad526476c45ffc6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 051b59f528ea4879ad526476c45ffc6e already in progress: creating tablet
I20260926 01:55:43.015672 9340 raft_consensus.cc:359] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.016295 9340 raft_consensus.cc:385] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.016530 9340 raft_consensus.cc:740] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.018421 9340 consensus_queue.cc:260] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.022815 8534 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d541e514029640269428f8bf5156eb58" } new_table_name: "default.Table_bf355929f0d5467bb45b71424eb7155b" modify_external_catalogs: true
I20260926 01:55:43.025637 9340 ts_tablet_manager.cc:1434] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
W20260926 01:55:43.030728 8524 consensus_peers.cc:597] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba -> Peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Couldn't send request to peer f75ba46482de422ea23daf17aa100849. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260926 01:55:43.031340 9340 tablet_bootstrap.cc:492] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:43.038328 9339 tablet_bootstrap.cc:492] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:43.038873 9339 ts_tablet_manager.cc:1403] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.261s user 0.032s sys 0.045s
I20260926 01:55:43.044407 9339 raft_consensus.cc:359] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:43.045213 9339 raft_consensus.cc:385] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.045470 9339 raft_consensus.cc:740] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.046223 9339 consensus_queue.cc:260] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:43.048122 9339 ts_tablet_manager.cc:1434] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260926 01:55:43.049454 9340 tablet_bootstrap.cc:654] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.053066 9339 tablet_bootstrap.cc:492] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:43.071503 9353 tablet_replica.cc:333] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:43.072101 9339 tablet_bootstrap.cc:654] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.072356 9353 raft_consensus.cc:2243] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.072847 9353 raft_consensus.cc:2272] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.075103 8533 master_service.cc:960] ReplaceTablet: received request to replace tablet 051b59f528ea4879ad526476c45ffc6e from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:43.075781 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 10fae6a23dbd4ce4adb012817384f2e7
I20260926 01:55:43.078122 8531 catalog_manager.cc:6934] ReplaceTablet: tablet 10fae6a23dbd4ce4adb012817384f2e7 deleted and replaced by tablet 8dfff96cc29145e7a72213bb5ecca1ae
I20260926 01:55:43.081277 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 10fae6a23dbd4ce4adb012817384f2e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dfff96cc29145e7a72213bb5ecca1ae) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:43.082239 9351 tablet_replica.cc:333] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:43.082923 9351 raft_consensus.cc:2243] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.083380 9351 raft_consensus.cc:2272] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.085345 9351 ts_tablet_manager.cc:1916] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.089530 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_66f4ea34f6154f8e93c1d3f472e1bbbc" } modify_external_catalogs: true
I20260926 01:55:43.090142 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_66f4ea34f6154f8e93c1d3f472e1bbbc" } modify_external_catalogs: true
I20260926 01:55:43.098228 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_7a3fdb246bf74e87883c7520bfd35695"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:43.100131 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a3fdb246bf74e87883c7520bfd35695 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:43.100128 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b827807def43b39b91d61a6dfd9386" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:43.102200 9155 leader_election.cc:343] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:43.105238 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 10fae6a23dbd4ce4adb012817384f2e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dfff96cc29145e7a72213bb5ecca1ae) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:43.116156 9353 ts_tablet_manager.cc:1916] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.119971 9341 tablet_bootstrap.cc:492] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:43.120664 9341 ts_tablet_manager.cc:1403] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.203s user 0.010s sys 0.002s
I20260926 01:55:43.121510 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b827807def43b39b91d61a6dfd9386" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:43.122414 8907 raft_consensus.cc:2468] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:43.123265 9341 raft_consensus.cc:359] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.123920 9341 raft_consensus.cc:385] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.124194 9341 raft_consensus.cc:740] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:55:43.124195 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_1c1a0d3391224ba291d16b4e18b5d1c0 [id=c241dc156c114a4684de8b3280984f1a], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 03b827807def43b39b91d61a6dfd9386 consensus state has no leader
I20260926 01:55:43.124956 9341 consensus_queue.cc:260] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.133068 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet d49f4812f2b3457890a51214acf2abab (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]) successfully deleted
W20260926 01:55:43.145999 8539 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28bd8695a6b34585bdb1bdabf1bcc654 has no consensus state
I20260926 01:55:43.146734 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet 10fae6a23dbd4ce4adb012817384f2e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8dfff96cc29145e7a72213bb5ecca1ae) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:43.151083 9339 tablet_bootstrap.cc:492] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:43.151597 9339 ts_tablet_manager.cc:1403] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.099s user 0.016s sys 0.019s
I20260926 01:55:43.154222 9339 raft_consensus.cc:359] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.154901 9339 raft_consensus.cc:385] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.155166 9339 raft_consensus.cc:740] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.155853 9339 consensus_queue.cc:260] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.160043 9340 tablet_bootstrap.cc:492] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:43.160099 8536 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 1171186cc294417db439bedf0ca4a6a2
I20260926 01:55:43.160542 9340 ts_tablet_manager.cc:1403] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.134s user 0.022s sys 0.021s
I20260926 01:55:43.163864 9339 ts_tablet_manager.cc:1434] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20260926 01:55:43.164873 9339 tablet_bootstrap.cc:492] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:43.126178 9152 leader_election.cc:304] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:43.177695 9341 ts_tablet_manager.cc:1434] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.057s user 0.008s sys 0.000s
W20260926 01:55:43.178200 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 52979.3us
W20260926 01:55:43.178318 9003 leader_election.cc:343] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: f7dcb2358e434f36a831352105863807
I20260926 01:55:43.178773 9341 tablet_bootstrap.cc:492] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:43.180349 9347 raft_consensus.cc:2804] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:43.180714 9347 raft_consensus.cc:493] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:43.181044 9347 raft_consensus.cc:3060] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:43.188051 9353 ts_tablet_manager.cc:1929] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.188535 9353 log.cc:1199] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/7009b837b5ac466d837e1aacebe88713
I20260926 01:55:43.189177 9353 ts_tablet_manager.cc:1950] T 7009b837b5ac466d837e1aacebe88713 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
W20260926 01:55:43.193246 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 450bace8066646f3833bcb4149b625b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 450bace8066646f3833bcb4149b625b6 already in progress: creating tablet
I20260926 01:55:43.194173 9353 tablet_replica.cc:333] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:43.194895 9353 raft_consensus.cc:2243] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.195362 9353 raft_consensus.cc:2272] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.196444 9347 raft_consensus.cc:515] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.197307 9353 ts_tablet_manager.cc:1916] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.199503 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b827807def43b39b91d61a6dfd9386" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:43.200052 9073 raft_consensus.cc:3060] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:43.205572 9347 leader_election.cc:290] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:43.208019 9073 raft_consensus.cc:2468] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:43.209460 9155 leader_election.cc:304] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:43.211493 9353 ts_tablet_manager.cc:1929] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.211826 9347 raft_consensus.cc:2804] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:43.211974 9353 log.cc:1199] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/f7dcb2358e434f36a831352105863807
I20260926 01:55:43.212605 9353 ts_tablet_manager.cc:1950] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:43.214380 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet f7dcb2358e434f36a831352105863807 (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f]) successfully deleted
I20260926 01:55:43.214728 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b827807def43b39b91d61a6dfd9386" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:43.215298 8907 raft_consensus.cc:3060] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:43.219043 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8e5e15ce8e58425697a7f013c09c0c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:43.219177 9340 raft_consensus.cc:359] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.219820 9340 raft_consensus.cc:385] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.220058 9340 raft_consensus.cc:740] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:55:43.220131 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 051b59f528ea4879ad526476c45ffc6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 051b59f528ea4879ad526476c45ffc6e already in progress: creating tablet
W20260926 01:55:43.220880 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e5e15ce8e58425697a7f013c09c0c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:43.222018 9353 tablet_replica.cc:333] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:43.222754 9353 raft_consensus.cc:2243] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.223213 9353 raft_consensus.cc:2272] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.223289 9347 raft_consensus.cc:697] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:43.224555 9347 consensus_queue.cc:237] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.230033 8907 raft_consensus.cc:2468] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:55:43.233294 9006 leader_election.cc:343] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: f7dcb2358e434f36a831352105863807
I20260926 01:55:43.233860 9353 ts_tablet_manager.cc:1916] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.233788 9006 leader_election.cc:304] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:43.235670 9341 tablet_bootstrap.cc:654] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:43.237658 8843 consensus_peers.cc:597] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:43.241665 9352 ts_tablet_manager.cc:1929] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.242061 9352 log.cc:1199] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/dda6b1627dd846f08d717f154c3e23d9
I20260926 01:55:43.242753 9352 ts_tablet_manager.cc:1950] T dda6b1627dd846f08d717f154c3e23d9 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
W20260926 01:55:43.242690 9006 raft_consensus.cc:2686] T f7dcb2358e434f36a831352105863807 P d20f26c9295641ecbf77d290ca206bb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 01:55:43.246330 9340 consensus_queue.cc:260] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.256718 9353 ts_tablet_manager.cc:1929] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260926 01:55:43.256966 8843 consensus_peers.cc:597] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:43.257128 9353 log.cc:1199] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/10fae6a23dbd4ce4adb012817384f2e7
I20260926 01:55:43.257750 9353 ts_tablet_manager.cc:1950] T 10fae6a23dbd4ce4adb012817384f2e7 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:43.260241 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]) successfully deleted
I20260926 01:55:43.262894 9340 ts_tablet_manager.cc:1434] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.102s user 0.008s sys 0.000s
I20260926 01:55:43.264757 9340 tablet_bootstrap.cc:492] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:43.268936 9352 tablet_replica.cc:333] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
W20260926 01:55:43.269111 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 051b59f528ea4879ad526476c45ffc6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 051b59f528ea4879ad526476c45ffc6e already in progress: creating tablet
I20260926 01:55:43.269676 9352 raft_consensus.cc:2243] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.270159 9352 raft_consensus.cc:2272] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.271994 9352 ts_tablet_manager.cc:1916] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.273444 9339 tablet_bootstrap.cc:654] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.277575 9341 tablet_bootstrap.cc:492] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:43.278013 9341 ts_tablet_manager.cc:1403] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.099s user 0.011s sys 0.004s
I20260926 01:55:43.281626 9341 raft_consensus.cc:359] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.282371 9341 raft_consensus.cc:385] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.282675 9341 raft_consensus.cc:740] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.283331 9341 consensus_queue.cc:260] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:55:43.288707 8534 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d10be686d9045efaf384d326b356e0e has no consensus state
I20260926 01:55:43.288997 9340 tablet_bootstrap.cc:654] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.291249 8538 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet da247e00df5a4633a45b6fee77004587
I20260926 01:55:43.299753 9341 ts_tablet_manager.cc:1434] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260926 01:55:43.300771 9341 tablet_bootstrap.cc:492] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:43.311070 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_72cdae2a885f4338977c6aed96a2cdcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:43.311578 9340 tablet_bootstrap.cc:492] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
W20260926 01:55:43.313053 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72cdae2a885f4338977c6aed96a2cdcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:43.316926 9340 ts_tablet_manager.cc:1403] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.000s
I20260926 01:55:43.317636 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_5979881fe26743e1ba0d4a710dc150e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:43.319052 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet 4d092195015a45c49cdcb60e343ff787 from {username='slave'} at 127.0.0.1:42048
W20260926 01:55:43.320737 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5979881fe26743e1ba0d4a710dc150e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:43.321118 9339 tablet_bootstrap.cc:492] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:43.321542 9339 ts_tablet_manager.cc:1403] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.002s
I20260926 01:55:43.324188 9339 raft_consensus.cc:359] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.324863 9339 raft_consensus.cc:385] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.325124 9339 raft_consensus.cc:740] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.328176 8538 master_service.cc:960] ReplaceTablet: received request to replace tablet 9bee4f08f4e6472b9ec711b23c67f7de from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:43.330137 9341 tablet_bootstrap.cc:654] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.333811 9352 ts_tablet_manager.cc:1929] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.334216 9352 log.cc:1199] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/7009b837b5ac466d837e1aacebe88713
I20260926 01:55:43.334883 9352 ts_tablet_manager.cc:1950] T 7009b837b5ac466d837e1aacebe88713 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:43.334511 9340 raft_consensus.cc:359] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.335330 9340 raft_consensus.cc:385] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.335587 9340 raft_consensus.cc:740] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.336350 9340 consensus_queue.cc:260] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.338817 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 7009b837b5ac466d837e1aacebe88713 (table default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc]) successfully deleted
I20260926 01:55:43.339828 9352 tablet_replica.cc:333] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:43.340510 9352 raft_consensus.cc:2243] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:43.341326 9352 pending_rounds.cc:70] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Trying to abort 2 pending ops.
I20260926 01:55:43.341346 9339 consensus_queue.cc:260] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.341534 9352 pending_rounds.cc:77] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333428196160040960 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "10fae6a23dbd4ce4adb012817384f2e7" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 2 new_table_name: "default.Table_d7233ccd12524b5eae620cd58c593335" dest_uuid: "49bd320bf73f4252804ec867ec077510" new_extra_config { } }
I20260926 01:55:43.342731 9340 ts_tablet_manager.cc:1434] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260926 01:55:43.343683 9340 tablet_bootstrap.cc:492] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:55:43.342744 9352 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58226: Aborted: Op aborted
W20260926 01:55:43.347461 8524 catalog_manager.cc:5142] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): alter failed for tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]): Aborted: Op aborted
I20260926 01:55:43.351450 9351 ts_tablet_manager.cc:1929] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.352216 9351 log.cc:1199] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/10fae6a23dbd4ce4adb012817384f2e7
I20260926 01:55:43.354472 8534 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_7a3fdb246bf74e87883c7520bfd35695" } new_table_name: "default.Table_8969d22a4ab54a5484838555a31bb684" modify_external_catalogs: true
I20260926 01:55:43.359588 9339 ts_tablet_manager.cc:1434] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20260926 01:55:43.363480 9352 pending_rounds.cc:77] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428193936367616 op_type: NO_OP noop_request { }
I20260926 01:55:43.364617 9352 raft_consensus.cc:2889] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:43.365087 9352 raft_consensus.cc:2272] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.379679 9339 tablet_bootstrap.cc:492] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:43.379752 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_1c1a0d3391224ba291d16b4e18b5d1c0 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 3 got 0
I20260926 01:55:43.383299 8539 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_1c1a0d3391224ba291d16b4e18b5d1c0 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 3 got 0
I20260926 01:55:43.383798 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 9d10be686d9045efaf384d326b356e0e (DEFAULT_TABLE table=default.Table_bf355929f0d5467bb45b71424eb7155b [id=2be262701c0b465f960fc3d666a78026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:43.385703 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 (DEFAULT_TABLE table=default.Table_27cdeab15da2426d85cbbb43441e39d6 [id=3b732759201f4eaab4fdf76cd8a38f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:43.387029 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28bd8695a6b34585bdb1bdabf1bcc654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:43.389572 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 (DEFAULT_TABLE table=default.Table_27cdeab15da2426d85cbbb43441e39d6 [id=3b732759201f4eaab4fdf76cd8a38f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:43.390417 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet dda6b1627dd846f08d717f154c3e23d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1171186cc294417db439bedf0ca4a6a2) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:43.392787 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d10be686d9045efaf384d326b356e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:43.377982 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 9d10be686d9045efaf384d326b356e0e (DEFAULT_TABLE table=default.Table_bf355929f0d5467bb45b71424eb7155b [id=2be262701c0b465f960fc3d666a78026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:43.399678 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d10be686d9045efaf384d326b356e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:43.384227 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 (DEFAULT_TABLE table=default.Table_27cdeab15da2426d85cbbb43441e39d6 [id=3b732759201f4eaab4fdf76cd8a38f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:43.403226 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28bd8695a6b34585bdb1bdabf1bcc654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:43.403780 9352 ts_tablet_manager.cc:1916] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.411448 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28bd8695a6b34585bdb1bdabf1bcc654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:43.411882 9340 tablet_bootstrap.cc:654] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.423023 9351 ts_tablet_manager.cc:1950] T 10fae6a23dbd4ce4adb012817384f2e7 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:43.426236 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]) successfully deleted
I20260926 01:55:43.428088 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 9d10be686d9045efaf384d326b356e0e (DEFAULT_TABLE table=default.Table_bf355929f0d5467bb45b71424eb7155b [id=2be262701c0b465f960fc3d666a78026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:43.429795 8536 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_27cdeab15da2426d85cbbb43441e39d6" } modify_external_catalogs: true
I20260926 01:55:43.430400 8536 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_27cdeab15da2426d85cbbb43441e39d6" } modify_external_catalogs: true
I20260926 01:55:43.445180 9349 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:44589,127.8.76.189:34163,127.8.76.188:38143
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:38143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.76.188
--webserver_port=33837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:55:43.447425 9349 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:55:43.438717 9340 tablet_bootstrap.cc:492] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:43.464329 9340 ts_tablet_manager.cc:1403] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.121s user 0.017s sys 0.000s
I20260926 01:55:43.466920 9340 raft_consensus.cc:359] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.467567 9340 raft_consensus.cc:385] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.467799 9340 raft_consensus.cc:740] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.443470 9352 ts_tablet_manager.cc:1929] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:43.468372 9352 log.cc:1199] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/10fae6a23dbd4ce4adb012817384f2e7
I20260926 01:55:43.469383 9352 ts_tablet_manager.cc:1950] T 10fae6a23dbd4ce4adb012817384f2e7 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:43.472466 9341 tablet_bootstrap.cc:492] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:43.472132 9340 consensus_queue.cc:260] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.472954 9341 ts_tablet_manager.cc:1403] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.172s user 0.035s sys 0.036s
I20260926 01:55:43.473620 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]) successfully deleted
W20260926 01:55:43.475533 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet dda6b1627dd846f08d717f154c3e23d9 because the tablet was not found. No further retry: Not found: Tablet not found: dda6b1627dd846f08d717f154c3e23d9
I20260926 01:55:43.477464 9339 tablet_bootstrap.cc:654] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.480499 9340 ts_tablet_manager.cc:1434] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260926 01:55:43.481494 9340 tablet_bootstrap.cc:492] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:43.486516 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d10be686d9045efaf384d326b356e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:43.493389 8535 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_6a7da4b61e7c428a8763e54c7ee9c870" } new_table_name: "default.Table_370cc280a4a6451cb1dae27ab9fce101" modify_external_catalogs: true
I20260926 01:55:43.495289 8533 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_e13d9ec3694c418e906e55984e87a263" } modify_external_catalogs: true
I20260926 01:55:43.495932 8533 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_e13d9ec3694c418e906e55984e87a263" } modify_external_catalogs: true
I20260926 01:55:43.496564 9345 raft_consensus.cc:493] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:43.497036 9345 raft_consensus.cc:515] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.499087 9345 leader_election.cc:290] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:43.499529 9345 raft_consensus.cc:493] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:43.499908 9345 raft_consensus.cc:515] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.504544 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "450bace8066646f3833bcb4149b625b6" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:43.504631 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bee4f08f4e6472b9ec711b23c67f7de" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:43.505167 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "450bace8066646f3833bcb4149b625b6" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:43.505870 9219 raft_consensus.cc:2468] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
W20260926 01:55:43.506306 8843 leader_election.cc:343] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:43.506809 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bee4f08f4e6472b9ec711b23c67f7de" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:55:43.507010 8843 leader_election.cc:343] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:43.507359 9220 raft_consensus.cc:2468] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:43.509718 8843 leader_election.cc:304] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:43.511672 9340 tablet_bootstrap.cc:654] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.527693 9341 raft_consensus.cc:359] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.528439 9341 raft_consensus.cc:385] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.528702 9341 raft_consensus.cc:740] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.529349 9341 consensus_queue.cc:260] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.535991 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b21204e6c05840bcb5d33a353d15d738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:43.537837 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b21204e6c05840bcb5d33a353d15d738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:43.542620 8534 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a85e47c1bf9f4e96958aa2de41039180 has no consensus state
I20260926 01:55:43.545678 9349 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260926 01:55:43.562435 8843 leader_election.cc:304] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:43.563521 9341 ts_tablet_manager.cc:1434] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.090s user 0.006s sys 0.001s
I20260926 01:55:43.564560 9341 tablet_bootstrap.cc:492] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:43.502259 9345 leader_election.cc:290] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:43.577054 9345 raft_consensus.cc:2804] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:43.577457 9345 raft_consensus.cc:493] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:43.577750 9345 raft_consensus.cc:3060] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:43.602873 9424 raft_consensus.cc:2804] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:43.603461 9424 raft_consensus.cc:493] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:43.604601 9424 raft_consensus.cc:3060] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:43.606769 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 44422.3us
I20260926 01:55:43.617390 9341 tablet_bootstrap.cc:654] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.620525 9339 tablet_bootstrap.cc:492] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:43.621007 9339 ts_tablet_manager.cc:1403] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.242s user 0.002s sys 0.031s
I20260926 01:55:43.623521 9339 raft_consensus.cc:359] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.624217 9339 raft_consensus.cc:385] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.624478 9339 raft_consensus.cc:740] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.625219 9339 consensus_queue.cc:260] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.634698 9339 ts_tablet_manager.cc:1434] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260926 01:55:43.635311 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 4d092195015a45c49cdcb60e343ff787
I20260926 01:55:43.636896 8531 catalog_manager.cc:6934] ReplaceTablet: tablet 4d092195015a45c49cdcb60e343ff787 deleted and replaced by tablet 5a0dacc61ff441788ff7c6b1aaf765fa
I20260926 01:55:43.638239 8536 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 28bd8695a6b34585bdb1bdabf1bcc654
I20260926 01:55:43.642525 9339 tablet_bootstrap.cc:492] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:43.647075 9347 raft_consensus.cc:493] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:43.647601 9347 raft_consensus.cc:515] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.649632 9347 leader_election.cc:290] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:43.651350 8533 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 099af935249a4310b29852a258661794
I20260926 01:55:43.652025 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:43.652103 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d092195015a45c49cdcb60e343ff787 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a0dacc61ff441788ff7c6b1aaf765fa) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:43.653514 9351 tablet_replica.cc:333] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:43.654197 9351 raft_consensus.cc:2243] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260926 01:55:43.655112 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 28bd8695a6b34585bdb1bdabf1bcc654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28bd8695a6b34585bdb1bdabf1bcc654 already in progress: creating tablet
I20260926 01:55:43.655540 9351 raft_consensus.cc:2272] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.656148 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:43.657371 9351 ts_tablet_manager.cc:1916] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.659096 8538 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 9bee4f08f4e6472b9ec711b23c67f7de
I20260926 01:55:43.660610 8538 catalog_manager.cc:6934] ReplaceTablet: tablet 9bee4f08f4e6472b9ec711b23c67f7de deleted and replaced by tablet 03086ec5451743fdbf2d5726d43f31b1
I20260926 01:55:43.662147 8540 catalog_manager.cc:5838] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:43.663697 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_1c1a0d3391224ba291d16b4e18b5d1c0 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 3 got 0
I20260926 01:55:43.668267 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "051b59f528ea4879ad526476c45ffc6e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:43.669341 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:43.669486 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d092195015a45c49cdcb60e343ff787 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a0dacc61ff441788ff7c6b1aaf765fa) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:43.669363 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:43.670831 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bee4f08f4e6472b9ec711b23c67f7de with delete_type TABLET_DATA_DELETED (replaced by tablet 03086ec5451743fdbf2d5726d43f31b1) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:43.672158 9155 leader_election.cc:343] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:43.672976 9353 tablet_replica.cc:333] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:43.673655 9353 raft_consensus.cc:2243] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.674048 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bee4f08f4e6472b9ec711b23c67f7de with delete_type TABLET_DATA_DELETED (replaced by tablet 03086ec5451743fdbf2d5726d43f31b1) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:43.674108 9353 raft_consensus.cc:2272] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:43.675462 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 28bd8695a6b34585bdb1bdabf1bcc654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28bd8695a6b34585bdb1bdabf1bcc654 already in progress: creating tablet
I20260926 01:55:43.676501 9353 ts_tablet_manager.cc:1916] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.685400 9339 tablet_bootstrap.cc:654] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:43.700726 9433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:43.704074 9353 ts_tablet_manager.cc:1929] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.704537 9353 log.cc:1199] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/4d092195015a45c49cdcb60e343ff787
I20260926 01:55:43.707625 9339 tablet_bootstrap.cc:492] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:43.708114 9339 ts_tablet_manager.cc:1403] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.000s
I20260926 01:55:43.713012 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "051b59f528ea4879ad526476c45ffc6e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:43.713668 9345 raft_consensus.cc:515] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:55:43.714838 9152 leader_election.cc:343] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:43.714660 9339 raft_consensus.cc:359] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.715327 9339 raft_consensus.cc:385] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.715312 9152 leader_election.cc:304] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:43.715574 9339 raft_consensus.cc:740] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.716475 9347 raft_consensus.cc:2749] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:43.716346 9339 consensus_queue.cc:260] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.717285 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "450bace8066646f3833bcb4149b625b6" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:43.717558 9345 leader_election.cc:290] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:43.717864 9220 raft_consensus.cc:3060] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:43.719600 9340 tablet_bootstrap.cc:492] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:43.720058 9340 ts_tablet_manager.cc:1403] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.239s user 0.031s sys 0.019s
I20260926 01:55:43.723248 9351 ts_tablet_manager.cc:1929] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.723596 9339 ts_tablet_manager.cc:1434] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260926 01:55:43.723486 9340 raft_consensus.cc:359] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.724176 9340 raft_consensus.cc:385] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.724428 9340 raft_consensus.cc:740] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.724608 9339 tablet_bootstrap.cc:492] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:43.730310 9353 ts_tablet_manager.cc:1950] T 4d092195015a45c49cdcb60e343ff787 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:43.733330 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 4d092195015a45c49cdcb60e343ff787 (table default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]) successfully deleted
I20260926 01:55:43.733512 9353 tablet_replica.cc:333] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:43.734180 9353 raft_consensus.cc:2243] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260926 01:55:43.734704 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 099af935249a4310b29852a258661794 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 099af935249a4310b29852a258661794 already in progress: creating tablet
I20260926 01:55:43.734885 9340 consensus_queue.cc:260] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.737999 9351 log.cc:1199] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/4d092195015a45c49cdcb60e343ff787
I20260926 01:55:43.741514 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "450bace8066646f3833bcb4149b625b6" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:43.726703 9220 raft_consensus.cc:2468] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:43.742113 9073 raft_consensus.cc:3060] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:43.743614 8843 leader_election.cc:304] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:43.744756 9345 raft_consensus.cc:2804] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:43.745225 9345 raft_consensus.cc:697] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:43.746025 9345 consensus_queue.cc:237] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.753966 9073 raft_consensus.cc:2468] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:43.756673 9351 ts_tablet_manager.cc:1950] T 4d092195015a45c49cdcb60e343ff787 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:43.759034 9340 ts_tablet_manager.cc:1434] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20260926 01:55:43.760027 9340 tablet_bootstrap.cc:492] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:43.764008 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_5979881fe26743e1ba0d4a710dc150e8" } new_table_name: "default.Table_265612f8bc2f48f9bba41010d7de8ee0" modify_external_catalogs: true
I20260926 01:55:43.770203 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 4d092195015a45c49cdcb60e343ff787 (table default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]) successfully deleted
I20260926 01:55:43.772394 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b42748cf16964c5eb69d2e72ba905f6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:43.773509 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_269f661dac8f41d08eebbdc0cd122fee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:43.775349 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_269f661dac8f41d08eebbdc0cd122fee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:43.776252 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_ff2478486a954f6e98d980bb68a0205c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:43.777906 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff2478486a954f6e98d980bb68a0205c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:43.783694 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 099af935249a4310b29852a258661794 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 099af935249a4310b29852a258661794 already in progress: creating tablet
W20260926 01:55:43.774319 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b42748cf16964c5eb69d2e72ba905f6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:43.785176 9351 tablet_replica.cc:333] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:43.763906 9353 raft_consensus.cc:2272] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.787225 9351 raft_consensus.cc:2243] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.789559 9351 raft_consensus.cc:2272] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.794338 9351 ts_tablet_manager.cc:1916] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.795028 9339 tablet_bootstrap.cc:654] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.804137 9353 ts_tablet_manager.cc:1916] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.811714 9340 tablet_bootstrap.cc:654] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.813007 9339 tablet_bootstrap.cc:492] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:43.813452 9339 ts_tablet_manager.cc:1403] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.000s
I20260926 01:55:43.814081 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d092195015a45c49cdcb60e343ff787 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a0dacc61ff441788ff7c6b1aaf765fa) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:43.815797 9352 tablet_replica.cc:333] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:43.816079 9339 raft_consensus.cc:359] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.816538 9352 raft_consensus.cc:2243] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.816720 9339 raft_consensus.cc:385] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.816998 9352 raft_consensus.cc:2272] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.816982 9339 raft_consensus.cc:740] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.818423 9341 tablet_bootstrap.cc:492] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:43.818890 9341 ts_tablet_manager.cc:1403] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.255s user 0.019s sys 0.019s
I20260926 01:55:43.820825 9352 ts_tablet_manager.cc:1916] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.821172 9341 raft_consensus.cc:359] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.821790 9341 raft_consensus.cc:385] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.822026 9341 raft_consensus.cc:740] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.826386 9341 consensus_queue.cc:260] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.831732 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:43.833276 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:43.834656 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bee4f08f4e6472b9ec711b23c67f7de with delete_type TABLET_DATA_DELETED (replaced by tablet 03086ec5451743fdbf2d5726d43f31b1) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:43.838786 9339 consensus_queue.cc:260] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.847918 8531 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_bf355929f0d5467bb45b71424eb7155b" } modify_external_catalogs: true
I20260926 01:55:43.848511 8531 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_bf355929f0d5467bb45b71424eb7155b" } modify_external_catalogs: true
I20260926 01:55:43.850673 9339 ts_tablet_manager.cc:1434] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
W20260926 01:55:43.855176 8535 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03086ec5451743fdbf2d5726d43f31b1 has no consensus state
I20260926 01:55:43.859685 9341 ts_tablet_manager.cc:1434] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260926 01:55:43.865923 9339 tablet_bootstrap.cc:492] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:43.872635 9341 tablet_bootstrap.cc:492] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:43.876313 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_37ba6de1661d4b1f95e93ea4b3348633" } modify_external_catalogs: true
I20260926 01:55:43.877004 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_37ba6de1661d4b1f95e93ea4b3348633" } modify_external_catalogs: true
I20260926 01:55:43.882350 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 051b59f528ea4879ad526476c45ffc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:43.883708 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 450bace8066646f3833bcb4149b625b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:43.890092 9353 ts_tablet_manager.cc:1929] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.890575 9353 log.cc:1199] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/9bee4f08f4e6472b9ec711b23c67f7de
I20260926 01:55:43.890630 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet 450bace8066646f3833bcb4149b625b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:43.891281 9353 ts_tablet_manager.cc:1950] T 9bee4f08f4e6472b9ec711b23c67f7de P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:43.894593 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 9bee4f08f4e6472b9ec711b23c67f7de (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]) successfully deleted
I20260926 01:55:43.895593 9353 tablet_replica.cc:333] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:43.896256 9353 raft_consensus.cc:2243] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.896715 9353 raft_consensus.cc:2272] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.897156 9341 tablet_bootstrap.cc:654] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.896268 9424 raft_consensus.cc:515] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.890151 9351 ts_tablet_manager.cc:1929] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.900101 9351 log.cc:1199] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/9bee4f08f4e6472b9ec711b23c67f7de
I20260926 01:55:43.900763 9351 ts_tablet_manager.cc:1950] T 9bee4f08f4e6472b9ec711b23c67f7de P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:55:43.901007 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 65658.1us
W20260926 01:55:43.902683 8524 catalog_manager.cc:5136] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): alter failed for tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]),no further retry: Not found: Tablet not found: 10fae6a23dbd4ce4adb012817384f2e7
I20260926 01:55:43.903980 9353 ts_tablet_manager.cc:1916] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.900398 9340 tablet_bootstrap.cc:492] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:43.904477 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 450bace8066646f3833bcb4149b625b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:43.908993 8843 leader_election.cc:343] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 9bee4f08f4e6472b9ec711b23c67f7de
I20260926 01:55:43.909210 9340 ts_tablet_manager.cc:1403] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.015s
I20260926 01:55:43.911676 9351 tablet_replica.cc:333] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:43.912474 9351 raft_consensus.cc:2243] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.912360 9340 raft_consensus.cc:359] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.913002 9340 raft_consensus.cc:385] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.913280 9340 raft_consensus.cc:740] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.912947 9351 raft_consensus.cc:2272] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:43.915506 9351 ts_tablet_manager.cc:1916] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:43.917959 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 9bee4f08f4e6472b9ec711b23c67f7de (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]) successfully deleted
I20260926 01:55:43.918083 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 9bee4f08f4e6472b9ec711b23c67f7de from {username='slave'} at 127.0.0.1:42048
W20260926 01:55:43.920817 8843 leader_election.cc:343] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 9bee4f08f4e6472b9ec711b23c67f7de
I20260926 01:55:43.921240 8843 leader_election.cc:304] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:43.921691 9340 consensus_queue.cc:260] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.922076 9345 raft_consensus.cc:2749] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:55:43.923166 9424 leader_election.cc:290] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:43.925319 8534 master_service.cc:960] ReplaceTablet: received request to replace tablet 9bee4f08f4e6472b9ec711b23c67f7de from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:43.928323 9339 tablet_bootstrap.cc:654] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.945070 9340 ts_tablet_manager.cc:1434] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.036s user 0.003s sys 0.004s
I20260926 01:55:43.947944 9341 tablet_bootstrap.cc:492] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:43.948562 9340 tablet_bootstrap.cc:492] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:43.948705 9341 ts_tablet_manager.cc:1403] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.076s user 0.013s sys 0.001s
I20260926 01:55:43.951063 8907 raft_consensus.cc:1275] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Refusing update from remote peer bc00feec3f564608851f0112e97ab532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:55:43.951391 9341 raft_consensus.cc:359] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.952005 9341 raft_consensus.cc:385] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.952234 9341 raft_consensus.cc:740] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.953308 9347 consensus_queue.cc:1048] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:43.953567 9073 raft_consensus.cc:1275] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Refusing update from remote peer bc00feec3f564608851f0112e97ab532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:55:43.955446 9347 consensus_queue.cc:1048] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:43.964555 9339 tablet_bootstrap.cc:492] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:43.965379 9339 ts_tablet_manager.cc:1403] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.100s user 0.013s sys 0.000s
I20260926 01:55:43.967978 9339 raft_consensus.cc:359] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.968680 9339 raft_consensus.cc:385] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:43.968954 9339 raft_consensus.cc:740] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:43.969694 9339 consensus_queue.cc:260] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:43.972249 9339 ts_tablet_manager.cc:1434] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:55:43.973090 9339 tablet_bootstrap.cc:492] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:43.974392 9340 tablet_bootstrap.cc:654] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:43.980739 9353 ts_tablet_manager.cc:1929] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.981153 9353 log.cc:1199] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/051b59f528ea4879ad526476c45ffc6e
I20260926 01:55:43.981773 9353 ts_tablet_manager.cc:1950] T 051b59f528ea4879ad526476c45ffc6e P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:43.974922 9341 consensus_queue.cc:260] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:43.984285 9353 tablet_replica.cc:333] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:43.984615 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 051b59f528ea4879ad526476c45ffc6e (table default.Table_6bcb6c89d7ba499e90436ceab11c20b9 [id=def17c340f58498cb4282a7b8599d730]) successfully deleted
I20260926 01:55:43.977356 9351 ts_tablet_manager.cc:1929] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:43.992358 9351 log.cc:1199] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/450bace8066646f3833bcb4149b625b6
I20260926 01:55:43.994910 9351 ts_tablet_manager.cc:1950] T 450bace8066646f3833bcb4149b625b6 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:43.997550 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 450bace8066646f3833bcb4149b625b6 (table default.Table_9dba2c1067df4cf397890976a54f09cd [id=fae5d0ce53564f6ca4b731ecbb660a55]) successfully deleted
I20260926 01:55:43.998191 9353 raft_consensus.cc:2243] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:43.998744 9353 raft_consensus.cc:2272] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:43.999291 8536 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a54eb3b8cf64a478df38df9d4aef02f has no consensus state
I20260926 01:55:44.000568 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_14869892bcf7429385632e2df181743c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:44.002269 9341 ts_tablet_manager.cc:1434] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.053s user 0.005s sys 0.001s
I20260926 01:55:44.012403 9353 ts_tablet_manager.cc:1916] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:44.012576 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14869892bcf7429385632e2df181743c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:44.025568 9352 ts_tablet_manager.cc:1929] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:44.035295 9341 tablet_bootstrap.cc:492] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:44.037261 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 9d10be686d9045efaf384d326b356e0e
I20260926 01:55:44.039304 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:44.040695 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:44.044209 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:44.052605 9341 tablet_bootstrap.cc:654] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:44.051908 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 9d10be686d9045efaf384d326b356e0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d10be686d9045efaf384d326b356e0e already in progress: creating tablet
I20260926 01:55:44.039724 9339 tablet_bootstrap.cc:654] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.028853 9352 log.cc:1199] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/4d092195015a45c49cdcb60e343ff787
I20260926 01:55:44.065631 9352 ts_tablet_manager.cc:1950] T 4d092195015a45c49cdcb60e343ff787 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:44.067030 9341 tablet_bootstrap.cc:492] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:44.067451 9341 ts_tablet_manager.cc:1403] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20260926 01:55:44.073688 9341 raft_consensus.cc:359] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.074373 9341 raft_consensus.cc:385] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.079953 9339 tablet_bootstrap.cc:492] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:44.080396 9339 ts_tablet_manager.cc:1403] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.000s
I20260926 01:55:44.083361 9339 raft_consensus.cc:359] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.084065 9339 raft_consensus.cc:385] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.084339 9339 raft_consensus.cc:740] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.074656 9341 raft_consensus.cc:740] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.085635 9341 consensus_queue.cc:260] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.092132 9341 ts_tablet_manager.cc:1434] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260926 01:55:44.093053 9341 tablet_bootstrap.cc:492] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:44.097259 9353 ts_tablet_manager.cc:1929] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:44.098526 9353 log.cc:1199] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/450bace8066646f3833bcb4149b625b6
I20260926 01:55:44.102414 9353 ts_tablet_manager.cc:1950] T 450bace8066646f3833bcb4149b625b6 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:44.102906 9339 consensus_queue.cc:260] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.108290 9339 ts_tablet_manager.cc:1434] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260926 01:55:44.111824 9339 tablet_bootstrap.cc:492] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:44.119545 9448 mvcc.cc:204] Tried to move back new op lower bound from 7333428199181959168 to 7333428196263550976. Current Snapshot: MvccSnapshot[applied={T|T < 7333428199181959168}]
I20260926 01:55:44.125609 9341 tablet_bootstrap.cc:654] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.131428 9339 tablet_bootstrap.cc:654] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.129240 9459 tablet.cc:1751] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:44.129827 9458 tablet.cc:1751] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:44.135730 9449 mvcc.cc:204] Tried to move back new op lower bound from 7333428199181959168 to 7333428196263550976. Current Snapshot: MvccSnapshot[applied={T|T < 7333428199181959168}]
I20260926 01:55:44.146144 9340 tablet_bootstrap.cc:492] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.146943 9340 ts_tablet_manager.cc:1403] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.201s user 0.019s sys 0.023s
I20260926 01:55:44.153683 9460 tablet.cc:1751] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:44.156818 9352 tablet_replica.cc:333] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:44.157626 9352 raft_consensus.cc:2243] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:44.158149 9352 raft_consensus.cc:2272] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:44.158310 9340 raft_consensus.cc:359] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.158936 9340 raft_consensus.cc:385] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.159188 9340 raft_consensus.cc:740] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.160182 9352 ts_tablet_manager.cc:1916] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:44.163862 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a85e47c1bf9f4e96958aa2de41039180 has no consensus state
I20260926 01:55:44.164952 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 4d092195015a45c49cdcb60e343ff787 (table default.Table_d482559c4505477fa6df7f7baab2e03f [id=f325cb544c7d4782862b6d10b9ca55ad]) successfully deleted
W20260926 01:55:44.163882 8525 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 03086ec5451743fdbf2d5726d43f31b1 has no consensus state
I20260926 01:55:44.166064 9341 tablet_bootstrap.cc:492] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:44.166738 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 450bace8066646f3833bcb4149b625b6 (table default.Table_9dba2c1067df4cf397890976a54f09cd [id=fae5d0ce53564f6ca4b731ecbb660a55]) successfully deleted
W20260926 01:55:44.167893 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 9d10be686d9045efaf384d326b356e0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d10be686d9045efaf384d326b356e0e already in progress: creating tablet
W20260926 01:55:44.166230 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 28bd8695a6b34585bdb1bdabf1bcc654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28bd8695a6b34585bdb1bdabf1bcc654 already in progress: creating tablet
I20260926 01:55:44.168895 9341 ts_tablet_manager.cc:1403] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.076s user 0.015s sys 0.000s
I20260926 01:55:44.174299 8532 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_265612f8bc2f48f9bba41010d7de8ee0" } new_table_name: "default.Table_292982f73b37414ebefe3d3493c525c5" modify_external_catalogs: true
W20260926 01:55:44.179850 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 099af935249a4310b29852a258661794 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 099af935249a4310b29852a258661794 already in progress: creating tablet
I20260926 01:55:44.164227 9339 tablet_bootstrap.cc:492] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:44.185293 9339 ts_tablet_manager.cc:1403] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.074s user 0.012s sys 0.003s
I20260926 01:55:44.186095 9352 ts_tablet_manager.cc:1929] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:44.186491 9352 log.cc:1199] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/9bee4f08f4e6472b9ec711b23c67f7de
I20260926 01:55:44.187127 9352 ts_tablet_manager.cc:1950] T 9bee4f08f4e6472b9ec711b23c67f7de P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:44.188175 9339 raft_consensus.cc:359] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.188817 9339 raft_consensus.cc:385] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.189069 9339 raft_consensus.cc:740] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.185987 9341 raft_consensus.cc:359] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.190022 9352 tablet_replica.cc:333] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:44.189826 9339 consensus_queue.cc:260] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.190006 9341 raft_consensus.cc:385] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.192780 9352 raft_consensus.cc:2243] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:44.191563 9341 raft_consensus.cc:740] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.193385 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 9bee4f08f4e6472b9ec711b23c67f7de (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]) successfully deleted
I20260926 01:55:44.193444 9352 pending_rounds.cc:70] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Trying to abort 1 pending ops.
I20260926 01:55:44.194263 9352 pending_rounds.cc:77] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428198391660544 op_type: NO_OP noop_request { }
I20260926 01:55:44.194630 9352 raft_consensus.cc:2889] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:44.194962 9352 raft_consensus.cc:2272] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:44.195699 9340 consensus_queue.cc:260] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.197814 9352 ts_tablet_manager.cc:1916] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:44.196170 9341 consensus_queue.cc:260] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.199702 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 1618eb529bcc4947b4c0e1e5504af71a (table default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]). Expected version 1 got 0
I20260926 01:55:44.204159 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_1c1a0d3391224ba291d16b4e18b5d1c0" } new_table_name: "default.Table_3951d686ab954ed784d5921048abb4a5" modify_external_catalogs: true
I20260926 01:55:44.216305 9341 ts_tablet_manager.cc:1434] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.045s user 0.009s sys 0.000s
I20260926 01:55:44.216886 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_d45fa6bcc6b14d1984b1e756819371a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:44.218972 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d45fa6bcc6b14d1984b1e756819371a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:44.223946 9341 tablet_bootstrap.cc:492] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:44.229221 9050 tablet_service.cc:1511] Processing CreateTablet for tablet a85e47c1bf9f4e96958aa2de41039180 (DEFAULT_TABLE table=default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.229301 9199 tablet_service.cc:1511] Processing CreateTablet for tablet a85e47c1bf9f4e96958aa2de41039180 (DEFAULT_TABLE table=default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.230507 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85e47c1bf9f4e96958aa2de41039180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.230504 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85e47c1bf9f4e96958aa2de41039180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.232123 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 051b59f528ea4879ad526476c45ffc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:44.232342 9352 ts_tablet_manager.cc:1929] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:44.232681 9352 log.cc:1199] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/450bace8066646f3833bcb4149b625b6
I20260926 01:55:44.232910 9351 tablet_replica.cc:333] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:44.233376 9352 ts_tablet_manager.cc:1950] T 450bace8066646f3833bcb4149b625b6 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:44.233588 9351 raft_consensus.cc:2243] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:44.234045 9351 raft_consensus.cc:2272] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:44.235275 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8fa5adfde54e424d935ea9df94631389"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:44.235890 9351 ts_tablet_manager.cc:1916] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:44.237124 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fa5adfde54e424d935ea9df94631389 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:44.237879 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 450bace8066646f3833bcb4149b625b6 (table default.Table_9dba2c1067df4cf397890976a54f09cd [id=fae5d0ce53564f6ca4b731ecbb660a55]) successfully deleted
W20260926 01:55:44.238672 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 9d10be686d9045efaf384d326b356e0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d10be686d9045efaf384d326b356e0e already in progress: creating tablet
I20260926 01:55:44.239905 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_cc558351862b4a39bc56a3883cba12ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:44.241591 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc558351862b4a39bc56a3883cba12ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:44.243464 8531 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_1c1a0d3391224ba291d16b4e18b5d1c0 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 3 got 0
I20260926 01:55:44.250552 8539 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d482559c4505477fa6df7f7baab2e03f" } modify_external_catalogs: true
I20260926 01:55:44.251187 8539 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d482559c4505477fa6df7f7baab2e03f" } modify_external_catalogs: true
I20260926 01:55:44.252270 9340 ts_tablet_manager.cc:1434] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.105s user 0.005s sys 0.000s
I20260926 01:55:44.254099 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 051b59f528ea4879ad526476c45ffc6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:44.254860 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 450bace8066646f3833bcb4149b625b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:42 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:44.255605 9352 tablet_replica.cc:333] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:44.255504 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d092195015a45c49cdcb60e343ff787 with delete_type TABLET_DATA_DELETED (replaced by tablet 5a0dacc61ff441788ff7c6b1aaf765fa) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:44.256317 9352 raft_consensus.cc:2243] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:44.256755 9352 raft_consensus.cc:2272] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:44.258638 9352 ts_tablet_manager.cc:1916] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:44.268497 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 3695411d251847b4b1ab0a1eb936a55a (DEFAULT_TABLE table=default.Table_37ba6de1661d4b1f95e93ea4b3348633 [id=a2b0b9ebe53d44bd99b0eae816b717fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.269712 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3695411d251847b4b1ab0a1eb936a55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.271245 8883 tablet_service.cc:1511] Processing CreateTablet for tablet a85e47c1bf9f4e96958aa2de41039180 (DEFAULT_TABLE table=default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.270359 9340 tablet_bootstrap.cc:492] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.272317 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85e47c1bf9f4e96958aa2de41039180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.274873 9352 ts_tablet_manager.cc:1929] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:44.275218 9352 log.cc:1199] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/051b59f528ea4879ad526476c45ffc6e
I20260926 01:55:44.275938 9352 ts_tablet_manager.cc:1950] T 051b59f528ea4879ad526476c45ffc6e P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:44.278692 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 051b59f528ea4879ad526476c45ffc6e (table default.Table_6bcb6c89d7ba499e90436ceab11c20b9 [id=def17c340f58498cb4282a7b8599d730]) successfully deleted
I20260926 01:55:44.279289 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_efe5a3dc3242481484fedd388b7be098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:44.279968 9341 tablet_bootstrap.cc:654] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:44.280383 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 450bace8066646f3833bcb4149b625b6 because the tablet was not found. No further retry: Not found: Tablet not found: 450bace8066646f3833bcb4149b625b6
W20260926 01:55:44.281016 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efe5a3dc3242481484fedd388b7be098 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:44.281976 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 4d092195015a45c49cdcb60e343ff787 because the tablet was not found. No further retry: Not found: Tablet not found: 4d092195015a45c49cdcb60e343ff787
I20260926 01:55:44.285171 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 3695411d251847b4b1ab0a1eb936a55a (DEFAULT_TABLE table=default.Table_37ba6de1661d4b1f95e93ea4b3348633 [id=a2b0b9ebe53d44bd99b0eae816b717fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.286369 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3695411d251847b4b1ab0a1eb936a55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.288643 9340 tablet_bootstrap.cc:654] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.293123 9445 mvcc.cc:204] Tried to move back new op lower bound from 7333428199181959168 to 7333428196263550976. Current Snapshot: MvccSnapshot[applied={T|T < 7333428199181959168}]
I20260926 01:55:44.300505 9459 tablet.cc:1746] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Skipping requested alter to schema version 3, tablet already version 3
I20260926 01:55:44.330277 9341 tablet_bootstrap.cc:492] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:44.330760 9341 ts_tablet_manager.cc:1403] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.108s user 0.011s sys 0.004s
I20260926 01:55:44.333238 9341 raft_consensus.cc:359] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.333899 9341 raft_consensus.cc:385] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.334152 9341 raft_consensus.cc:740] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.334383 9458 tablet.cc:1746] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Skipping requested alter to schema version 3, tablet already version 3
I20260926 01:55:44.334875 9341 consensus_queue.cc:260] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.335263 8884 tablet_service.cc:1511] Processing CreateTablet for tablet a601d850375c43679dde7a46cfe71d22 (DEFAULT_TABLE table=default.Table_caaf07e0f7d24f7388b358362fd1668f [id=6b7b2329cccd4e96acfc168a00029171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.336400 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a601d850375c43679dde7a46cfe71d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.344419 9200 tablet_service.cc:1511] Processing CreateTablet for tablet a601d850375c43679dde7a46cfe71d22 (DEFAULT_TABLE table=default.Table_caaf07e0f7d24f7388b358362fd1668f [id=6b7b2329cccd4e96acfc168a00029171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.345417 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a601d850375c43679dde7a46cfe71d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.346647 9341 ts_tablet_manager.cc:1434] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260926 01:55:44.351825 9341 tablet_bootstrap.cc:492] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:44.359628 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 1618eb529bcc4947b4c0e1e5504af71a (table default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]). Expected version 1 got 0
I20260926 01:55:44.368348 9340 tablet_bootstrap.cc:492] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.368820 9340 ts_tablet_manager.cc:1403] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.099s user 0.024s sys 0.006s
I20260926 01:55:44.371906 9340 raft_consensus.cc:359] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.373008 9340 raft_consensus.cc:385] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.373436 9340 raft_consensus.cc:740] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.374802 9341 tablet_bootstrap.cc:654] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.377331 9340 consensus_queue.cc:260] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.387264 9339 ts_tablet_manager.cc:1434] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.202s user 0.006s sys 0.000s
I20260926 01:55:44.388638 9339 tablet_bootstrap.cc:492] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:44.390606 9367 raft_consensus.cc:493] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.391072 9367 raft_consensus.cc:515] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.393118 9367 leader_election.cc:290] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:44.396651 8885 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfff96cc29145e7a72213bb5ecca1ae (DEFAULT_TABLE table=default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.397806 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfff96cc29145e7a72213bb5ecca1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.401376 9340 ts_tablet_manager.cc:1434] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.032s user 0.008s sys 0.002s
I20260926 01:55:44.405184 9465 raft_consensus.cc:493] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.405036 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 3695411d251847b4b1ab0a1eb936a55a (DEFAULT_TABLE table=default.Table_37ba6de1661d4b1f95e93ea4b3348633 [id=a2b0b9ebe53d44bd99b0eae816b717fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.405736 9465 raft_consensus.cc:515] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.407218 9051 tablet_service.cc:1511] Processing CreateTablet for tablet a601d850375c43679dde7a46cfe71d22 (DEFAULT_TABLE table=default.Table_caaf07e0f7d24f7388b358362fd1668f [id=6b7b2329cccd4e96acfc168a00029171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.408080 9465 leader_election.cc:290] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:44.409415 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:44.409960 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:44.411566 9152 leader_election.cc:343] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:44.412071 9155 leader_election.cc:343] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:44.412389 9006 leader_election.cc:343] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: dda6b1627dd846f08d717f154c3e23d9
I20260926 01:55:44.412453 9155 leader_election.cc:304] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:44.413232 9465 raft_consensus.cc:2749] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:44.413673 9340 tablet_bootstrap.cc:492] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.430356 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfff96cc29145e7a72213bb5ecca1ae (DEFAULT_TABLE table=default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:44.432898 8524 catalog_manager.cc:5136] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): alter failed for tablet 10fae6a23dbd4ce4adb012817384f2e7 (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]),no further retry: Not found: Tablet not found: 10fae6a23dbd4ce4adb012817384f2e7
I20260926 01:55:44.406287 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3695411d251847b4b1ab0a1eb936a55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.435155 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a601d850375c43679dde7a46cfe71d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.435333 9200 tablet_service.cc:1511] Processing CreateTablet for tablet 8dfff96cc29145e7a72213bb5ecca1ae (DEFAULT_TABLE table=default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:44.436434 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfff96cc29145e7a72213bb5ecca1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:44.438886 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfff96cc29145e7a72213bb5ecca1ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:44.443239 9003 leader_election.cc:343] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: dda6b1627dd846f08d717f154c3e23d9
I20260926 01:55:44.449246 9340 tablet_bootstrap.cc:654] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.470659 9003 leader_election.cc:304] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:44.472342 9339 tablet_bootstrap.cc:654] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.473068 9367 raft_consensus.cc:2749] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:44.475559 9351 ts_tablet_manager.cc:1929] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:44.475980 9351 log.cc:1199] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/051b59f528ea4879ad526476c45ffc6e
I20260926 01:55:44.476631 9351 ts_tablet_manager.cc:1950] T 051b59f528ea4879ad526476c45ffc6e P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:44.485267 9340 tablet_bootstrap.cc:492] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.485781 9340 ts_tablet_manager.cc:1403] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.079s user 0.011s sys 0.001s
I20260926 01:55:44.491789 9465 raft_consensus.cc:493] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.492421 9465 raft_consensus.cc:515] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.494760 9465 leader_election.cc:290] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:44.496415 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1618eb529bcc4947b4c0e1e5504af71a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:44.496150 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1618eb529bcc4947b4c0e1e5504af71a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:44.497228 8907 raft_consensus.cc:2468] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:44.498603 9152 leader_election.cc:304] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
W20260926 01:55:44.499370 9155 leader_election.cc:343] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:44.499799 9465 raft_consensus.cc:2804] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:44.500129 9465 raft_consensus.cc:493] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:44.494415 9340 raft_consensus.cc:359] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.500454 9465 raft_consensus.cc:3060] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.500864 9340 raft_consensus.cc:385] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.501197 9340 raft_consensus.cc:740] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.501953 9340 consensus_queue.cc:260] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.504330 9340 ts_tablet_manager.cc:1434] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20260926 01:55:44.505298 9340 tablet_bootstrap.cc:492] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.509481 9465 raft_consensus.cc:515] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.514945 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 051b59f528ea4879ad526476c45ffc6e (table default.Table_6bcb6c89d7ba499e90436ceab11c20b9 [id=def17c340f58498cb4282a7b8599d730]) successfully deleted
I20260926 01:55:44.520337 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1618eb529bcc4947b4c0e1e5504af71a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:44.520951 8907 raft_consensus.cc:3060] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.522208 9340 tablet_bootstrap.cc:654] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.525476 9339 tablet_bootstrap.cc:492] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:44.525938 9339 ts_tablet_manager.cc:1403] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.002s
I20260926 01:55:44.528604 9339 raft_consensus.cc:359] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.529299 9339 raft_consensus.cc:385] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.529152 8907 raft_consensus.cc:2468] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:44.529568 9339 raft_consensus.cc:740] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.531208 9152 leader_election.cc:304] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:44.533305 9347 raft_consensus.cc:2804] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:44.533808 9347 raft_consensus.cc:697] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:44.534608 9347 consensus_queue.cc:237] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.540679 9340 tablet_bootstrap.cc:492] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.541237 9340 ts_tablet_manager.cc:1403] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.003s
I20260926 01:55:44.544050 9340 raft_consensus.cc:359] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.544802 9340 raft_consensus.cc:385] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.545064 9340 raft_consensus.cc:740] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.544678 9339 consensus_queue.cc:260] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.547016 9339 ts_tablet_manager.cc:1434] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.021s user 0.003s sys 0.005s
I20260926 01:55:44.552079 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1618eb529bcc4947b4c0e1e5504af71a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:44.552688 9073 raft_consensus.cc:3060] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.552766 9465 leader_election.cc:290] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:44.545805 9340 consensus_queue.cc:260] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.566942 9340 ts_tablet_manager.cc:1434] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
W20260926 01:55:44.566890 8532 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a54eb3b8cf64a478df38df9d4aef02f has no consensus state
I20260926 01:55:44.569617 9340 tablet_bootstrap.cc:492] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.571959 8538 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 3695411d251847b4b1ab0a1eb936a55a
I20260926 01:55:44.574107 8538 master_service.cc:960] ReplaceTablet: received request to replace tablet 1618eb529bcc4947b4c0e1e5504af71a from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:44.574712 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:44.576115 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:44.577472 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:44.578881 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 3695411d251847b4b1ab0a1eb936a55a because the tablet was not found. No further retry: Not found: Tablet not found: 3695411d251847b4b1ab0a1eb936a55a
W20260926 01:55:44.579612 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 3695411d251847b4b1ab0a1eb936a55a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3695411d251847b4b1ab0a1eb936a55a already in progress: creating tablet
W20260926 01:55:44.581632 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 3695411d251847b4b1ab0a1eb936a55a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3695411d251847b4b1ab0a1eb936a55a already in progress: creating tablet
I20260926 01:55:44.593636 9340 tablet_bootstrap.cc:654] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.611922 9340 tablet_bootstrap.cc:492] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.612437 9340 ts_tablet_manager.cc:1403] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.004s
I20260926 01:55:44.622045 9340 raft_consensus.cc:359] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.622810 9340 raft_consensus.cc:385] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.623070 9340 raft_consensus.cc:740] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.632655 9340 consensus_queue.cc:260] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.634281 9367 raft_consensus.cc:493] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.634868 9367 raft_consensus.cc:515] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.635504 9424 raft_consensus.cc:493] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.635983 9424 raft_consensus.cc:515] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.637899 9424 leader_election.cc:290] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:44.639982 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:44.640643 9220 raft_consensus.cc:2468] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:44.642164 8843 leader_election.cc:304] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:44.643132 9424 raft_consensus.cc:2804] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:44.643509 9424 raft_consensus.cc:493] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:44.644148 9424 raft_consensus.cc:3060] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.653182 9424 raft_consensus.cc:515] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.656347 9340 ts_tablet_manager.cc:1434] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.044s user 0.009s sys 0.000s
I20260926 01:55:44.659415 9340 tablet_bootstrap.cc:492] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.659971 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:44.660535 9220 raft_consensus.cc:3060] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.668509 9220 raft_consensus.cc:2468] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:44.670173 8843 leader_election.cc:304] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:44.672740 9424 leader_election.cc:290] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:44.675580 9424 raft_consensus.cc:2804] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:44.676689 9424 raft_consensus.cc:697] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:44.678541 8533 master_service.cc:960] ReplaceTablet: received request to replace tablet 28bd8695a6b34585bdb1bdabf1bcc654 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:44.681782 9424 consensus_queue.cc:237] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.679860 9340 tablet_bootstrap.cc:654] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.701723 9340 tablet_bootstrap.cc:492] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.702191 9340 ts_tablet_manager.cc:1403] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.001s
I20260926 01:55:44.702926 9459 tablet.cc:1751] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:55:44.704599 8539 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 5a0dacc61ff441788ff7c6b1aaf765fa
I20260926 01:55:44.710479 9458 tablet.cc:1751] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:55:44.726064 9340 raft_consensus.cc:359] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.726804 9340 raft_consensus.cc:385] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.727058 9340 raft_consensus.cc:740] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.727698 9340 consensus_queue.cc:260] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.729579 9340 ts_tablet_manager.cc:1434] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.027s user 0.002s sys 0.003s
I20260926 01:55:44.731784 9367 leader_election.cc:290] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:44.732446 9339 tablet_bootstrap.cc:492] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:44.732348 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:44.732921 9460 tablet.cc:1746] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Skipping requested alter to schema version 3, tablet already version 3
I20260926 01:55:44.734548 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:44.734824 8843 leader_election.cc:343] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:44.735805 8843 leader_election.cc:343] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:44.738788 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4411120140f3410c9f4e33e26da20d98" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:44.739646 9220 raft_consensus.cc:2468] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:44.740491 9073 raft_consensus.cc:2468] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:44.740931 9006 leader_election.cc:304] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:44.741946 9367 raft_consensus.cc:2804] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:44.742290 9367 raft_consensus.cc:493] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:44.742625 9367 raft_consensus.cc:3060] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.730877 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4411120140f3410c9f4e33e26da20d98" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:44.745741 8907 raft_consensus.cc:2468] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:44.761197 9340 tablet_bootstrap.cc:492] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.768414 9340 tablet_bootstrap.cc:654] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.766335 9339 tablet_bootstrap.cc:654] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.778144 9424 raft_consensus.cc:493] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.780511 9424 raft_consensus.cc:515] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.785356 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:44.786105 9073 raft_consensus.cc:2468] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:44.787744 8843 leader_election.cc:304] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:44.788015 9367 raft_consensus.cc:515] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.790974 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:44.791306 9424 leader_election.cc:290] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:44.791678 9220 raft_consensus.cc:2468] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:44.792704 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4411120140f3410c9f4e33e26da20d98" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:44.793301 8907 raft_consensus.cc:3060] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.797626 9340 tablet_bootstrap.cc:492] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.798086 9340 ts_tablet_manager.cc:1403] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.003s
I20260926 01:55:44.791852 9424 raft_consensus.cc:2804] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:44.800068 9424 raft_consensus.cc:493] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:44.800518 9424 raft_consensus.cc:3060] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.800637 9340 raft_consensus.cc:359] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:44.802345 8907 raft_consensus.cc:2468] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:44.804113 9003 leader_election.cc:304] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:44.805311 9340 raft_consensus.cc:385] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.805644 9340 raft_consensus.cc:740] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.805987 9474 tablet.cc:1751] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:55:44.806507 9480 raft_consensus.cc:493] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.806383 9340 consensus_queue.cc:260] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:44.807163 9480 raft_consensus.cc:515] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.808522 9340 ts_tablet_manager.cc:1434] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260926 01:55:44.809540 9340 tablet_bootstrap.cc:492] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.809944 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4411120140f3410c9f4e33e26da20d98" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:44.810606 9220 raft_consensus.cc:3060] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:44.818209 9220 raft_consensus.cc:2468] T 4411120140f3410c9f4e33e26da20d98 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:44.825218 9341 tablet_bootstrap.cc:492] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:44.812186 9482 raft_consensus.cc:2804] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:44.825708 9341 ts_tablet_manager.cc:1403] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.478s user 0.055s sys 0.051s
I20260926 01:55:44.828356 9341 raft_consensus.cc:359] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.829471 9341 raft_consensus.cc:385] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.829746 9341 raft_consensus.cc:740] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.830696 9341 consensus_queue.cc:260] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.842834 9341 ts_tablet_manager.cc:1434] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.017s user 0.003s sys 0.005s
I20260926 01:55:44.844439 9339 tablet_bootstrap.cc:492] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:44.844916 9339 ts_tablet_manager.cc:1403] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.280s user 0.011s sys 0.020s
I20260926 01:55:44.845352 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_14869892bcf7429385632e2df181743c" } modify_external_catalogs: true
I20260926 01:55:44.845943 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_14869892bcf7429385632e2df181743c" } modify_external_catalogs: true
I20260926 01:55:44.847098 9341 tablet_bootstrap.cc:492] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:44.848613 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_ff2478486a954f6e98d980bb68a0205c" } new_table_name: "default.Table_e96413b8ff0340f9b3698862eb4525fe" modify_external_catalogs: true
I20260926 01:55:44.850164 9339 raft_consensus.cc:359] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.850816 9339 raft_consensus.cc:385] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.852680 9341 tablet_bootstrap.cc:654] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.834290 9367 leader_election.cc:290] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:44.857035 9340 tablet_bootstrap.cc:654] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.858263 9480 leader_election.cc:290] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:44.851066 9339 raft_consensus.cc:740] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.859998 9339 consensus_queue.cc:260] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.867033 9339 ts_tablet_manager.cc:1434] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260926 01:55:44.869218 9339 tablet_bootstrap.cc:492] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:44.876825 9340 tablet_bootstrap.cc:492] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:44.877202 9482 raft_consensus.cc:697] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:44.877269 9340 ts_tablet_manager.cc:1403] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.000s
I20260926 01:55:44.878435 9482 consensus_queue.cc:237] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.880156 9340 raft_consensus.cc:359] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:44.880797 9340 raft_consensus.cc:385] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.881042 9340 raft_consensus.cc:740] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.881666 9340 consensus_queue.cc:260] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:44.884582 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_523e24048b654f60bf78f4f9821423c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:44.886855 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_523e24048b654f60bf78f4f9821423c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:44.891108 9475 raft_consensus.cc:493] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:44.891721 9475 raft_consensus.cc:515] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.895897 9475 leader_election.cc:290] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:44.896659 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "099af935249a4310b29852a258661794" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:44.897341 9220 raft_consensus.cc:2468] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:44.897683 9340 ts_tablet_manager.cc:1434] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260926 01:55:44.898674 9340 tablet_bootstrap.cc:492] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:44.898880 8843 leader_election.cc:304] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:44.901146 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_b21204e6c05840bcb5d33a353d15d738" } new_table_name: "default.Table_e1303d8861b344649b96fbfb1377d81b" modify_external_catalogs: true
I20260926 01:55:44.907058 9339 tablet_bootstrap.cc:654] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.907732 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_f0d2c43d4a9146e792b26da571026dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:44.909643 9341 tablet_bootstrap.cc:492] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
W20260926 01:55:44.909552 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0d2c43d4a9146e792b26da571026dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:44.910089 9341 ts_tablet_manager.cc:1403] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.004s
I20260926 01:55:44.912688 9341 raft_consensus.cc:359] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.913106 8536 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d25a39f7e9fa4682af9a020d33a580ad" } modify_external_catalogs: true
I20260926 01:55:44.913352 9341 raft_consensus.cc:385] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:44.913584 9341 raft_consensus.cc:740] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.913682 8536 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d25a39f7e9fa4682af9a020d33a580ad" } modify_external_catalogs: true
I20260926 01:55:44.914338 9341 consensus_queue.cc:260] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:44.928336 9341 ts_tablet_manager.cc:1434] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260926 01:55:44.930754 9340 tablet_bootstrap.cc:654] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:44.936983 9341 tablet_bootstrap.cc:492] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:44.944221 9424 raft_consensus.cc:515] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:44.957508 9341 tablet_bootstrap.cc:654] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:44.967312 8539 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac2e6e544aae4ac3a2d3cc116d22e581 has no consensus state
W20260926 01:55:44.996048 8540 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04e79a2e52fb40619b2bc99396abda5b has no consensus state
I20260926 01:55:45.003974 9341 tablet_bootstrap.cc:492] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:45.004446 9341 ts_tablet_manager.cc:1403] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.004s
I20260926 01:55:45.007197 9341 raft_consensus.cc:359] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.007869 9341 raft_consensus.cc:385] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.008119 9341 raft_consensus.cc:740] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.018081 9341 consensus_queue.cc:260] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.019958 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b0ac2f1cf5c54da5b9418e25248f4ebd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.021551 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0ac2f1cf5c54da5b9418e25248f4ebd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.023944 9341 ts_tablet_manager.cc:1434] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
W20260926 01:55:45.027122 8524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260926 01:55:45.027557 9341 tablet_bootstrap.cc:492] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:45.029376 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_286c6d4fb9b744348506faeeff2a3ba5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.031739 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_286c6d4fb9b744348506faeeff2a3ba5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.033576 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:45.034271 9220 raft_consensus.cc:2468] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:45.036686 9341 tablet_bootstrap.cc:654] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.036309 9006 leader_election.cc:304] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:45.038280 9482 raft_consensus.cc:2804] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:45.038980 9482 raft_consensus.cc:493] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.039351 9482 raft_consensus.cc:3060] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.040485 9339 tablet_bootstrap.cc:492] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:45.041311 9339 ts_tablet_manager.cc:1403] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.172s user 0.011s sys 0.000s
I20260926 01:55:45.045552 9339 raft_consensus.cc:359] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.046574 9339 raft_consensus.cc:385] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.046962 9339 raft_consensus.cc:740] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:44.973263 9340 tablet_bootstrap.cc:492] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:45.060747 9340 ts_tablet_manager.cc:1403] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.162s user 0.003s sys 0.008s
I20260926 01:55:45.063378 9340 raft_consensus.cc:359] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.064056 9340 raft_consensus.cc:385] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.064323 9340 raft_consensus.cc:740] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.065047 9340 consensus_queue.cc:260] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:44.999737 9475 raft_consensus.cc:2804] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:45.066172 9475 raft_consensus.cc:493] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.066577 9475 raft_consensus.cc:3060] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.006191 9494 raft_consensus.cc:493] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:45.068454 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:45.068997 9341 tablet_bootstrap.cc:492] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:45.069057 9220 raft_consensus.cc:3060] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.069487 9341 ts_tablet_manager.cc:1403] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.005s
I20260926 01:55:45.068341 9494 raft_consensus.cc:515] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.073482 9341 raft_consensus.cc:359] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.074076 9341 raft_consensus.cc:385] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.075235 9494 leader_election.cc:290] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.074347 9341 raft_consensus.cc:740] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.077294 9339 consensus_queue.cc:260] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.077718 9482 raft_consensus.cc:515] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.078047 9341 consensus_queue.cc:260] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.086977 9341 ts_tablet_manager.cc:1434] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260926 01:55:45.091487 9340 ts_tablet_manager.cc:1434] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20260926 01:55:45.094627 9341 tablet_bootstrap.cc:492] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:45.099454 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:55:45.081696 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 277537us
I20260926 01:55:45.102389 9494 raft_consensus.cc:493] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:45.104592 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a85e47c1bf9f4e96958aa2de41039180" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:45.105216 9219 raft_consensus.cc:2468] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:45.084884 9482 leader_election.cc:290] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.108403 9339 ts_tablet_manager.cc:1434] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.067s user 0.007s sys 0.006s
I20260926 01:55:45.108573 9341 tablet_bootstrap.cc:654] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.112531 9340 tablet_bootstrap.cc:492] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:45.115309 9339 tablet_bootstrap.cc:492] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:45.106312 9494 raft_consensus.cc:515] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.118705 9494 leader_election.cc:290] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:45.119168 9494 raft_consensus.cc:493] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:45.119541 9494 raft_consensus.cc:515] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.121270 9494 leader_election.cc:290] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:45.123034 8843 leader_election.cc:304] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
W20260926 01:55:45.125706 9049 consensus_meta.cc:287] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.573s user 0.010s sys 0.000s
I20260926 01:55:45.127183 9475 raft_consensus.cc:515] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.128437 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d10be686d9045efaf384d326b356e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:45.128825 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28bd8695a6b34585bdb1bdabf1bcc654" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:45.129101 9219 raft_consensus.cc:2468] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:45.129469 9218 raft_consensus.cc:2468] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:45.141515 9339 tablet_bootstrap.cc:654] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.131256 8843 leader_election.cc:304] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.132194 9494 raft_consensus.cc:2804] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:45.142818 9494 raft_consensus.cc:493] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.143306 9494 raft_consensus.cc:3060] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.146560 8843 leader_election.cc:304] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.148975 9475 leader_election.cc:290] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.155068 9475 raft_consensus.cc:2804] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:45.157246 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "099af935249a4310b29852a258661794" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:45.155169 9499 raft_consensus.cc:2804] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:45.157933 9218 raft_consensus.cc:3060] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.157994 9499 raft_consensus.cc:493] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.158427 9499 raft_consensus.cc:3060] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.164300 9494 raft_consensus.cc:515] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.167001 9341 tablet_bootstrap.cc:492] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:45.167584 9341 ts_tablet_manager.cc:1403] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.000s
I20260926 01:55:45.171737 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "099af935249a4310b29852a258661794" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:45.167829 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:45.172508 9073 raft_consensus.cc:2468] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:45.172851 8907 raft_consensus.cc:2393] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:55:45.169745 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:45.170142 9499 raft_consensus.cc:515] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.174045 8906 raft_consensus.cc:2393] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:55:45.157994 9475 raft_consensus.cc:493] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.174384 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d9cb4de16da44e1b90ac79e9e32cb88" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:45.174615 9475 raft_consensus.cc:3060] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.175153 9073 raft_consensus.cc:2393] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:45.170305 9341 raft_consensus.cc:359] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.176903 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a85e47c1bf9f4e96958aa2de41039180" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:45.177402 9341 raft_consensus.cc:385] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.179466 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d10be686d9045efaf384d326b356e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:45.183732 9499 leader_election.cc:290] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:45.179347 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a85e47c1bf9f4e96958aa2de41039180" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:45.182996 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28bd8695a6b34585bdb1bdabf1bcc654" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:45.184392 9219 raft_consensus.cc:3060] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.184407 9217 raft_consensus.cc:3060] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.186779 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28bd8695a6b34585bdb1bdabf1bcc654" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:45.187160 9475 raft_consensus.cc:515] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.188333 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "099af935249a4310b29852a258661794" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:45.188844 9073 raft_consensus.cc:3060] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.191736 9217 raft_consensus.cc:2468] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.192028 9219 raft_consensus.cc:2468] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.193938 9341 raft_consensus.cc:740] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.193807 8843 leader_election.cc:304] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.194407 9494 leader_election.cc:290] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.194767 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a85e47c1bf9f4e96958aa2de41039180" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:45.194778 9499 raft_consensus.cc:2804] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.195535 9499 raft_consensus.cc:697] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:45.195443 8843 leader_election.cc:304] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.195930 9220 raft_consensus.cc:2468] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.196342 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28bd8695a6b34585bdb1bdabf1bcc654" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:45.196566 9475 leader_election.cc:290] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:45.199986 9475 raft_consensus.cc:2804] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.197140 9499 consensus_queue.cc:237] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.200973 9341 consensus_queue.cc:260] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.201412 9003 leader_election.cc:304] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:55:45.202905 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 170235us
W20260926 01:55:45.203612 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a85e47c1bf9f4e96958aa2de41039180 consensus state has no leader
I20260926 01:55:45.201539 9218 raft_consensus.cc:2468] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.197060 9424 leader_election.cc:290] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.200449 9475 raft_consensus.cc:697] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:45.209729 9475 consensus_queue.cc:237] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.213471 9341 ts_tablet_manager.cc:1434] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260926 01:55:45.213719 9340 tablet_bootstrap.cc:654] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.213893 8539 master_service.cc:960] ReplaceTablet: received request to replace tablet 03b827807def43b39b91d61a6dfd9386 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:45.214941 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_f0d2c43d4a9146e792b26da571026dd0" } new_table_name: "default.Table_8c00cbced1a64f6c9568cedb86846d44" modify_external_catalogs: true
I20260926 01:55:45.200201 9494 raft_consensus.cc:493] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:45.221396 9494 raft_consensus.cc:515] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.226506 9341 tablet_bootstrap.cc:492] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:45.239336 9494 leader_election.cc:290] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:45.254633 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 59269.3us
W20260926 01:55:45.257658 8843 leader_election.cc:343] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:45.258605 8843 leader_election.cc:343] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:45.259434 8843 leader_election.cc:343] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.259943 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_214d8f2884b048ebba2f07a67d5effe8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.260288 8843 leader_election.cc:343] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:45.261094 8843 leader_election.cc:343] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:45.261919 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_214d8f2884b048ebba2f07a67d5effe8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.261961 8843 leader_election.cc:304] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:45.265018 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d10be686d9045efaf384d326b356e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:45.265630 9218 raft_consensus.cc:3060] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.267912 9339 tablet_bootstrap.cc:492] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:45.268389 9339 ts_tablet_manager.cc:1403] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.001s
I20260926 01:55:45.268395 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3695411d251847b4b1ab0a1eb936a55a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:45.269179 9220 raft_consensus.cc:2468] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
W20260926 01:55:45.271975 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32e0fe60b8744dbb8ae2bafefe3e834d has no consensus state
I20260926 01:55:45.272502 9218 raft_consensus.cc:2468] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.275418 9341 tablet_bootstrap.cc:654] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.277653 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8e3268d0d292489ea3b7a692f557954c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.280898 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e3268d0d292489ea3b7a692f557954c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.280982 9482 raft_consensus.cc:2749] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260926 01:55:45.282263 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 80945.7us
I20260926 01:55:45.283756 9339 raft_consensus.cc:359] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.284446 9339 raft_consensus.cc:385] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.284708 9339 raft_consensus.cc:740] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.286273 9339 consensus_queue.cc:260] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.287317 9073 raft_consensus.cc:2468] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.288966 8843 leader_election.cc:304] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.289789 9475 raft_consensus.cc:2804] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.290278 9475 raft_consensus.cc:697] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:45.290398 9494 raft_consensus.cc:2804] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.291169 9475 consensus_queue.cc:237] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.290854 9494 raft_consensus.cc:697] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:45.279201 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_b42748cf16964c5eb69d2e72ba905f6b" } new_table_name: "default.Table_d7957d605b514c30a979aa567bec903e" modify_external_catalogs: true
I20260926 01:55:45.312355 9341 tablet_bootstrap.cc:492] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:45.312829 9341 ts_tablet_manager.cc:1403] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.004s
I20260926 01:55:45.318549 9494 consensus_queue.cc:237] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.321013 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d10be686d9045efaf384d326b356e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:45.322751 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3695411d251847b4b1ab0a1eb936a55a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:45.332779 9339 ts_tablet_manager.cc:1434] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.064s user 0.007s sys 0.001s
I20260926 01:55:45.333853 9339 tablet_bootstrap.cc:492] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:45.289924 8843 leader_election.cc:304] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.355012 9475 raft_consensus.cc:2804] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260926 01:55:45.354912 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 65062.6us
I20260926 01:55:45.355376 9475 raft_consensus.cc:493] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.355680 9475 raft_consensus.cc:3060] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.358292 9339 tablet_bootstrap.cc:654] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.362521 9340 tablet_bootstrap.cc:492] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:45.362957 9340 ts_tablet_manager.cc:1403] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.251s user 0.005s sys 0.018s
I20260926 01:55:45.363282 9475 raft_consensus.cc:515] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.365468 9340 raft_consensus.cc:359] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.366096 9340 raft_consensus.cc:385] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.367597 9341 raft_consensus.cc:359] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.368278 9341 raft_consensus.cc:385] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.368527 9341 raft_consensus.cc:740] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.368577 9200 tablet_service.cc:1511] Processing CreateTablet for tablet 562604137d484e83a51de2f778d58fca (DEFAULT_TABLE table=default.Table_b42748cf16964c5eb69d2e72ba905f6b [id=b202eb85ec1944d8ad57f8521b37170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.369689 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 562604137d484e83a51de2f778d58fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.370189 9475 leader_election.cc:290] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.355698 8843 leader_election.cc:304] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.373486 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3695411d251847b4b1ab0a1eb936a55a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:55:45.373704 8843 leader_election.cc:343] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.374070 9218 raft_consensus.cc:3060] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.373754 8885 tablet_service.cc:1511] Processing CreateTablet for tablet 562604137d484e83a51de2f778d58fca (DEFAULT_TABLE table=default.Table_b42748cf16964c5eb69d2e72ba905f6b [id=b202eb85ec1944d8ad57f8521b37170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.380803 9218 raft_consensus.cc:2468] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.375864 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 562604137d484e83a51de2f778d58fca. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:45.375509 8843 leader_election.cc:343] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.382342 8843 leader_election.cc:304] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.383306 9501 raft_consensus.cc:2804] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.383742 9501 raft_consensus.cc:697] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:45.384541 9501 consensus_queue.cc:237] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.386758 9341 consensus_queue.cc:260] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.376037 9475 raft_consensus.cc:2804] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.391091 9475 raft_consensus.cc:697] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:45.393447 9475 consensus_queue.cc:237] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.402299 9341 ts_tablet_manager.cc:1434] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.089s user 0.007s sys 0.000s
I20260926 01:55:45.366355 9340 raft_consensus.cc:740] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.372553 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3695411d251847b4b1ab0a1eb936a55a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:55:45.406150 8843 leader_election.cc:343] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.362221 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 562604137d484e83a51de2f778d58fca (DEFAULT_TABLE table=default.Table_b42748cf16964c5eb69d2e72ba905f6b [id=b202eb85ec1944d8ad57f8521b37170b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.406800 9340 consensus_queue.cc:260] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.408789 9340 ts_tablet_manager.cc:1434] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.046s user 0.004s sys 0.000s
I20260926 01:55:45.409612 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 562604137d484e83a51de2f778d58fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.411759 9050 tablet_service.cc:1511] Processing CreateTablet for tablet cbf8818e65d74a32ba21dcb8a9a56a62 (DEFAULT_TABLE table=default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.413100 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf8818e65d74a32ba21dcb8a9a56a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.424229 9339 tablet_bootstrap.cc:492] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:45.424696 9339 ts_tablet_manager.cc:1403] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.006s
I20260926 01:55:45.427330 9339 raft_consensus.cc:359] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.427945 9339 raft_consensus.cc:385] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.428191 9339 raft_consensus.cc:740] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.432789 8882 tablet_service.cc:1511] Processing CreateTablet for tablet cbf8818e65d74a32ba21dcb8a9a56a62 (DEFAULT_TABLE table=default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.433485 9197 tablet_service.cc:1511] Processing CreateTablet for tablet cbf8818e65d74a32ba21dcb8a9a56a62 (DEFAULT_TABLE table=default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.434109 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf8818e65d74a32ba21dcb8a9a56a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.434576 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf8818e65d74a32ba21dcb8a9a56a62. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:45.440982 8524 consensus_peers.cc:597] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba -> Peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Couldn't send request to peer f75ba46482de422ea23daf17aa100849. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260926 01:55:45.441807 9339 consensus_queue.cc:260] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.455996 9465 consensus_queue.cc:1048] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:45.469396 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 (DEFAULT_TABLE table=default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.470549 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 (DEFAULT_TABLE table=default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.472785 9341 tablet_bootstrap.cc:492] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:45.474936 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 562604137d484e83a51de2f778d58fca consensus state has no leader
I20260926 01:55:45.475013 9340 tablet_bootstrap.cc:492] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:45.478209 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_4fdfd0dfce964d90b625381eabd6eac5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.480623 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fdfd0dfce964d90b625381eabd6eac5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.481503 9494 raft_consensus.cc:493] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:45.482009 9494 raft_consensus.cc:515] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.484936 9465 consensus_queue.cc:1048] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:45.487039 9501 consensus_queue.cc:1048] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:45.488749 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a601d850375c43679dde7a46cfe71d22" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:45.489398 9218 raft_consensus.cc:2468] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:45.492059 8843 leader_election.cc:304] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.492918 9501 raft_consensus.cc:2804] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:45.493244 9501 raft_consensus.cc:493] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.493506 9501 raft_consensus.cc:3060] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.487205 9494 leader_election.cc:290] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.507783 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a601d850375c43679dde7a46cfe71d22" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:45.515120 9052 tablet_service.cc:1511] Processing CreateTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 (DEFAULT_TABLE table=default.Table_14869892bcf7429385632e2df181743c [id=bfa8586ef17949bead9731402cd78042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.515372 9465 raft_consensus.cc:493] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:45.515591 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03086ec5451743fdbf2d5726d43f31b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.515885 9465 raft_consensus.cc:515] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.518023 9465 leader_election.cc:290] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:45.519024 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfff96cc29145e7a72213bb5ecca1ae" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:45.519697 8907 raft_consensus.cc:2468] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:45.522446 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 (DEFAULT_TABLE table=default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.527074 8538 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 1618eb529bcc4947b4c0e1e5504af71a
I20260926 01:55:45.528730 8538 catalog_manager.cc:6934] ReplaceTablet: tablet 1618eb529bcc4947b4c0e1e5504af71a deleted and replaced by tablet a09351c3db384b4383ca5ff0eaca982c
I20260926 01:55:45.529148 8538 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 91) took 1105 ms. Trace:
I20260926 01:55:45.530827 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03086ec5451743fdbf2d5726d43f31b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.533864 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a with delete_type TABLET_DATA_DELETED (replaced by tablet a09351c3db384b4383ca5ff0eaca982c) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:45.538733 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfff96cc29145e7a72213bb5ecca1ae" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:45.541280 9155 leader_election.cc:343] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.542047 9152 leader_election.cc:304] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:45.543171 9465 raft_consensus.cc:2804] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:45.543499 9465 raft_consensus.cc:493] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:45.543812 9465 raft_consensus.cc:3060] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.548420 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a with delete_type TABLET_DATA_DELETED (replaced by tablet a09351c3db384b4383ca5ff0eaca982c) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:45.554775 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a with delete_type TABLET_DATA_DELETED (replaced by tablet a09351c3db384b4383ca5ff0eaca982c) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:45.557503 9341 tablet_bootstrap.cc:654] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.559710 9516 tablet_replica.cc:333] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:45.560626 9516 raft_consensus.cc:2243] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:45.561833 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 4 got 3
I20260926 01:55:45.562402 9516 pending_rounds.cc:70] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Trying to abort 1 pending ops.
I20260926 01:55:45.562729 9516 pending_rounds.cc:77] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428201625812992 op_type: NO_OP noop_request { }
I20260926 01:55:45.563135 9516 raft_consensus.cc:2889] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:45.563511 9516 raft_consensus.cc:2272] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:45.565546 8531 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 4 got 3
I20260926 01:55:45.565778 9516 ts_tablet_manager.cc:1916] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:45.567998 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03086ec5451743fdbf2d5726d43f31b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.568580 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dccbfbc9fb5947f9b85ca8560e205b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.571883 8533 catalog_manager.cc:5838] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:45.573544 8533 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]). Expected version 1 got 0
I20260926 01:55:45.574304 8533 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 1 got 0
W20260926 01:55:45.598270 8843 leader_election.cc:343] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.598811 8885 tablet_service.cc:1511] Processing CreateTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 (DEFAULT_TABLE table=default.Table_14869892bcf7429385632e2df181743c [id=bfa8586ef17949bead9731402cd78042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.600860 9072 raft_consensus.cc:3060] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.601012 9200 tablet_service.cc:1511] Processing CreateTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 (DEFAULT_TABLE table=default.Table_14869892bcf7429385632e2df181743c [id=bfa8586ef17949bead9731402cd78042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.602173 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dccbfbc9fb5947f9b85ca8560e205b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.603983 9514 tablet_replica.cc:333] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:45.604379 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dccbfbc9fb5947f9b85ca8560e205b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.529508 8538 rpcz_store.cc:276] 0926 01:55:44.423958 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:44.573967 (+150009us) service_pool.cc:224] Handling call
0926 01:55:44.840593 (+266626us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60a0e1 00007f717c1d2d37 00007f717c1d2cc2 00007f717c1d2bf1 00007f717c1d2b6c 00007f717c1d1e22 00007f717a45c09c 00007f717a45b4d6
0926 01:55:45.529065 (+688472us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:45.448020 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:45.448492 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:45.448517 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:55:45.448529 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:45.448853 (+ 324us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:45.448893 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:45.448914 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:45.449128 (+ 214us) tablet.cc:701] Row locks acquired
0926 01:55:45.449141 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:45.449260 (+ 119us) write_op.cc:270] Start()
0926 01:55:45.449350 (+ 90us) write_op.cc:276] Timestamp: P: 1790387745448685 usec, L: 0
0926 01:55:45.449365 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:45.449876 (+ 511us) log.cc:844] Serialized 624 byte log entry
0926 01:55:45.522792 (+ 72916us) op_driver.cc:464] REPLICATION: finished
0926 01:55:45.524223 (+ 1431us) write_op.cc:301] APPLY: starting
0926 01:55:45.524286 (+ 63us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:45.524407 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:45.524507 (+ 100us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:45.524884 (+ 377us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:45.525076 (+ 192us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:45.525114 (+ 38us) write_op.cc:312] APPLY: finished
0926 01:55:45.525300 (+ 186us) log.cc:844] Serialized 28 byte log entry
0926 01:55:45.525431 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:45.525497 (+ 66us) write_op.cc:454] Released schema lock
0926 01:55:45.525528 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:45.525755 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:45.525777 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":568869120,"child_traces":[["op",{"apply.queue_time_us":1257,"apply.run_cpu_time_us":1499,"apply.run_wall_time_us":1498,"num_ops":2,"prepare.queue_time_us":95104,"prepare.run_cpu_time_us":2113,"prepare.run_wall_time_us":3313,"raft.queue_time_us":85,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":73312}]]}
I20260926 01:55:45.612244 9514 raft_consensus.cc:2243] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:45.612171 8535 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_286c6d4fb9b744348506faeeff2a3ba5" } new_table_name: "default.Table_9c4d9e672f6f409096edc1f1995e23e7" modify_external_catalogs: true
I20260926 01:55:45.618623 9514 pending_rounds.cc:70] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Trying to abort 1 pending ops.
I20260926 01:55:45.619045 9514 pending_rounds.cc:77] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428201625812992 op_type: NO_OP noop_request { }
I20260926 01:55:45.620218 9514 raft_consensus.cc:2889] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:45.620688 9514 raft_consensus.cc:2272] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:45.624297 9514 ts_tablet_manager.cc:1916] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:45.625447 8534 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:45.629292 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:45.630297 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 04e79a2e52fb40619b2bc99396abda5b (DEFAULT_TABLE table=default.Table_e1303d8861b344649b96fbfb1377d81b [id=2af856781c904c77a11ca9f4966ca44b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.631256 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e79a2e52fb40619b2bc99396abda5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.632349 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:45.634544 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 04e79a2e52fb40619b2bc99396abda5b (DEFAULT_TABLE table=default.Table_e1303d8861b344649b96fbfb1377d81b [id=2af856781c904c77a11ca9f4966ca44b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.635424 9340 tablet_bootstrap.cc:654] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.635560 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e79a2e52fb40619b2bc99396abda5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.635452 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f (DEFAULT_TABLE table=default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.636590 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a54eb3b8cf64a478df38df9d4aef02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.648131 9465 raft_consensus.cc:515] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.650686 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_48399c44f8254aaeaa623db43553e1ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.652604 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48399c44f8254aaeaa623db43553e1ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.669411 9339 ts_tablet_manager.cc:1434] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.244s user 0.005s sys 0.003s
I20260926 01:55:45.666448 9482 raft_consensus.cc:493] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:45.680001 9340 tablet_bootstrap.cc:492] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:45.680461 9340 ts_tablet_manager.cc:1403] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.206s user 0.011s sys 0.009s
I20260926 01:55:45.682637 9341 tablet_bootstrap.cc:492] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:45.646165 9501 raft_consensus.cc:515] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.683454 9341 ts_tablet_manager.cc:1403] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.211s user 0.018s sys 0.005s
I20260926 01:55:45.684465 9501 leader_election.cc:290] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.685704 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a601d850375c43679dde7a46cfe71d22" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:45.686987 9220 raft_consensus.cc:3060] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.686717 9341 raft_consensus.cc:359] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.687474 9341 raft_consensus.cc:385] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.687721 9341 raft_consensus.cc:740] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.688335 9341 consensus_queue.cc:260] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.686803 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f (DEFAULT_TABLE table=default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.691614 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a54eb3b8cf64a478df38df9d4aef02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.698544 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfff96cc29145e7a72213bb5ecca1ae" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:45.699272 8906 raft_consensus.cc:3060] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:45.703362 9220 raft_consensus.cc:2468] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:45.704887 8843 leader_election.cc:304] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.706525 8906 raft_consensus.cc:2468] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:45.707829 9152 leader_election.cc:304] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:45.709360 9424 raft_consensus.cc:2804] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.709908 9424 raft_consensus.cc:697] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:45.710776 9424 consensus_queue.cc:237] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:45.718259 8882 tablet_service.cc:1511] Processing CreateTablet for tablet b659eca3c97144aaacd41e747aecd96d (DEFAULT_TABLE table=default.Table_269f661dac8f41d08eebbdc0cd122fee [id=0dae284524224264b7ccbd59cd975030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.719628 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b659eca3c97144aaacd41e747aecd96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.677321 9482 raft_consensus.cc:515] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:45.734807 9482 leader_election.cc:290] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:45.737887 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 04e79a2e52fb40619b2bc99396abda5b (DEFAULT_TABLE table=default.Table_e1303d8861b344649b96fbfb1377d81b [id=2af856781c904c77a11ca9f4966ca44b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.739180 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e79a2e52fb40619b2bc99396abda5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.738112 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:45.741031 9339 tablet_bootstrap.cc:492] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:45.741137 9523 raft_consensus.cc:2804] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:45.742692 9523 raft_consensus.cc:697] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:45.742995 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f (DEFAULT_TABLE table=default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.744233 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a54eb3b8cf64a478df38df9d4aef02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.744127 9523 consensus_queue.cc:237] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.750059 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28bd8695a6b34585bdb1bdabf1bcc654" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:45.752525 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a601d850375c43679dde7a46cfe71d22" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:55:45.754904 8535 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01052a5e22a74a19919bae9dd0296291 has no consensus state
W20260926 01:55:45.759188 8843 leader_election.cc:343] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.763912 9515 tablet_replica.cc:333] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:45.764118 9053 tablet_service.cc:1511] Processing CreateTablet for tablet b659eca3c97144aaacd41e747aecd96d (DEFAULT_TABLE table=default.Table_269f661dac8f41d08eebbdc0cd122fee [id=0dae284524224264b7ccbd59cd975030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.764887 9515 raft_consensus.cc:2243] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:45.765259 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b659eca3c97144aaacd41e747aecd96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.765347 9515 pending_rounds.cc:70] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Trying to abort 1 pending ops.
I20260926 01:55:45.765615 9515 pending_rounds.cc:77] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428201625812992 op_type: NO_OP noop_request { }
W20260926 01:55:45.763885 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 56143.6us
I20260926 01:55:45.766057 9515 raft_consensus.cc:2889] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:45.766446 9515 raft_consensus.cc:2272] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:45.768726 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_e1303d8861b344649b96fbfb1377d81b" } new_table_name: "default.Table_8c9193cf386d4eca9776e2ceb2c7dc11" modify_external_catalogs: true
I20260926 01:55:45.769819 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_29ce250e9d9a4162966731d01a8e3b23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.773420 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29ce250e9d9a4162966731d01a8e3b23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.775389 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28bd8695a6b34585bdb1bdabf1bcc654" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:45.776298 9220 raft_consensus.cc:2393] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:55:45.779695 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfff96cc29145e7a72213bb5ecca1ae" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:45.789947 9197 tablet_service.cc:1511] Processing CreateTablet for tablet b659eca3c97144aaacd41e747aecd96d (DEFAULT_TABLE table=default.Table_269f661dac8f41d08eebbdc0cd122fee [id=0dae284524224264b7ccbd59cd975030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.791273 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b659eca3c97144aaacd41e747aecd96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.791049 8884 tablet_service.cc:1511] Processing CreateTablet for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (DEFAULT_TABLE table=default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.792140 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2e6e544aae4ac3a2d3cc116d22e581. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:45.798169 9155 leader_election.cc:343] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:45.802599 9341 ts_tablet_manager.cc:1434] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.119s user 0.008s sys 0.001s
I20260926 01:55:45.804090 9341 tablet_bootstrap.cc:492] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:45.819028 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_dd6c9b79a6a34f00818d8af57fd472bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:45.821007 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd6c9b79a6a34f00818d8af57fd472bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:45.823894 9340 raft_consensus.cc:359] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.824623 9340 raft_consensus.cc:385] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.824868 9340 raft_consensus.cc:740] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.825675 9340 consensus_queue.cc:260] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.835049 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8e3268d0d292489ea3b7a692f557954c" } modify_external_catalogs: true
I20260926 01:55:45.836512 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8e3268d0d292489ea3b7a692f557954c" } modify_external_catalogs: true
I20260926 01:55:45.788522 9465 leader_election.cc:290] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:45.884836 9516 ts_tablet_manager.cc:1929] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:45.891247 9516 log.cc:1199] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/1618eb529bcc4947b4c0e1e5504af71a
I20260926 01:55:45.906055 9516 ts_tablet_manager.cc:1950] T 1618eb529bcc4947b4c0e1e5504af71a P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:45.909302 9515 ts_tablet_manager.cc:1916] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:45.910627 9341 tablet_bootstrap.cc:654] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:45.911864 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 1618eb529bcc4947b4c0e1e5504af71a (table default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]) successfully deleted
W20260926 01:55:45.920462 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet dccbfbc9fb5947f9b85ca8560e205b08 because the tablet was not found. No further retry: Not found: Tablet not found: dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:45.921344 9529 consensus_queue.cc:1048] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:55:45.931699 9198 tablet_service.cc:1511] Processing CreateTablet for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (DEFAULT_TABLE table=default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.933092 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2e6e544aae4ac3a2d3cc116d22e581. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:45.936654 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 48404us
I20260926 01:55:45.938429 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 (DEFAULT_TABLE table=default.Table_8e5e15ce8e58425697a7f013c09c0c72 [id=5dd626e9121c4d5aba51637d7c6cb8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.940979 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1866cd97bb024a9baecf101a6d5b1ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.945248 9340 ts_tablet_manager.cc:1434] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.265s user 0.007s sys 0.000s
I20260926 01:55:45.946384 9514 ts_tablet_manager.cc:1929] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:45.949709 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 (DEFAULT_TABLE table=default.Table_8e5e15ce8e58425697a7f013c09c0c72 [id=5dd626e9121c4d5aba51637d7c6cb8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:45.951069 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1866cd97bb024a9baecf101a6d5b1ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:45.952463 9514 log.cc:1199] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/1618eb529bcc4947b4c0e1e5504af71a
I20260926 01:55:45.953372 9514 ts_tablet_manager.cc:1950] T 1618eb529bcc4947b4c0e1e5504af71a P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:45.960342 9341 tablet_bootstrap.cc:492] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:45.961007 9341 ts_tablet_manager.cc:1403] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.157s user 0.006s sys 0.011s
I20260926 01:55:45.963580 9341 raft_consensus.cc:359] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.964221 9341 raft_consensus.cc:385] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:45.964476 9341 raft_consensus.cc:740] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:45.967841 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 1618eb529bcc4947b4c0e1e5504af71a (table default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]) successfully deleted
I20260926 01:55:45.969412 9341 consensus_queue.cc:260] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:45.974516 9341 ts_tablet_manager.cc:1434] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
W20260926 01:55:45.982995 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet dccbfbc9fb5947f9b85ca8560e205b08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dccbfbc9fb5947f9b85ca8560e205b08 already in progress: creating tablet
I20260926 01:55:45.982590 9341 tablet_bootstrap.cc:492] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:45.990635 9340 tablet_bootstrap.cc:492] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:45.992877 9339 tablet_bootstrap.cc:654] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.006861 9341 tablet_bootstrap.cc:654] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.030623 9341 tablet_bootstrap.cc:492] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:46.031586 9341 ts_tablet_manager.cc:1403] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.000s
I20260926 01:55:46.034032 9341 raft_consensus.cc:359] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.034521 9341 raft_consensus.cc:385] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.034744 9341 raft_consensus.cc:740] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.035421 9341 consensus_queue.cc:260] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.040900 9482 raft_consensus.cc:493] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.041427 9482 raft_consensus.cc:515] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.044596 9424 consensus_queue.cc:1048] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:46.045547 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:46.044986 9006 leader_election.cc:304] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:46.046962 9529 raft_consensus.cc:3060] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:46.049950 9341 ts_tablet_manager.cc:1434] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260926 01:55:46.054960 9482 leader_election.cc:290] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:46.055644 9341 tablet_bootstrap.cc:492] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:46.056296 9340 tablet_bootstrap.cc:654] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.059569 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 (DEFAULT_TABLE table=default.Table_8e5e15ce8e58425697a7f013c09c0c72 [id=5dd626e9121c4d5aba51637d7c6cb8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.060798 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1866cd97bb024a9baecf101a6d5b1ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.064097 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a011a9edfe94e04afb6ea0367dd979f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:46.066298 9046 tablet_service.cc:1511] Processing CreateTablet for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (DEFAULT_TABLE table=default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.067505 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2e6e544aae4ac3a2d3cc116d22e581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.067423 8577 master_runner.cc:187] Error getting master registration for 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111),
I20260926 01:55:46.068415 8577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:55:46.070163 9482 consensus_queue.cc:1048] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:46.089881 9529 raft_consensus.cc:2749] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:46.098234 9341 tablet_bootstrap.cc:654] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.102142 9006 leader_election.cc:304] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:55:46.104717 8843 consensus_peers.cc:597] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:46.122023 9424 consensus_queue.cc:1048] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:46.131016 9341 tablet_bootstrap.cc:492] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:46.131475 9341 ts_tablet_manager.cc:1403] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.000s
I20260926 01:55:46.138044 9341 raft_consensus.cc:359] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.143185 9341 raft_consensus.cc:385] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.143476 9341 raft_consensus.cc:740] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.144259 9341 consensus_queue.cc:260] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.145686 8539 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 03b827807def43b39b91d61a6dfd9386
I20260926 01:55:46.147382 8539 catalog_manager.cc:6934] ReplaceTablet: tablet 03b827807def43b39b91d61a6dfd9386 deleted and replaced by tablet 70ef5265eb6244aa96c8195aa30cdfb0
I20260926 01:55:46.151955 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 03b827807def43b39b91d61a6dfd9386 with delete_type TABLET_DATA_DELETED (replaced by tablet 70ef5265eb6244aa96c8195aa30cdfb0) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.152755 9516 tablet_replica.cc:333] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:46.153522 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 03b827807def43b39b91d61a6dfd9386 with delete_type TABLET_DATA_DELETED (replaced by tablet 70ef5265eb6244aa96c8195aa30cdfb0) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:46.153646 9516 raft_consensus.cc:2243] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:46.154255 9514 tablet_replica.cc:333] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:46.154791 9516 raft_consensus.cc:2272] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.155133 9514 raft_consensus.cc:2243] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.155664 9514 raft_consensus.cc:2272] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.156798 9516 ts_tablet_manager.cc:1916] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:46.157629 9514 ts_tablet_manager.cc:1916] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:46.160703 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04e79a2e52fb40619b2bc99396abda5b consensus state has no leader
I20260926 01:55:46.162942 9341 ts_tablet_manager.cc:1434] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260926 01:55:46.167073 9339 tablet_bootstrap.cc:492] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:46.167512 9339 ts_tablet_manager.cc:1403] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.427s user 0.003s sys 0.014s
I20260926 01:55:46.169236 8538 master_service.cc:960] ReplaceTablet: received request to replace tablet 1618eb529bcc4947b4c0e1e5504af71a from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:46.173357 8534 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 204cf735a0114eb8aa242d2e1c6a2e42
I20260926 01:55:46.174551 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_2a36d6777b8c442d99c2cd88d9110efd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.176970 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_b0ac2f1cf5c54da5b9418e25248f4ebd" } new_table_name: "default.Table_c085afec1ab24b588fa3ff21bc9f5d96" modify_external_catalogs: true
I20260926 01:55:46.180781 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]). Expected version 1 got 0
I20260926 01:55:46.181694 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 1 got 0
I20260926 01:55:46.182390 8536 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 9d9cb4de16da44e1b90ac79e9e32cb88
I20260926 01:55:46.182659 9339 raft_consensus.cc:359] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.183313 9339 raft_consensus.cc:385] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.183594 9339 raft_consensus.cc:740] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.184182 8536 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 98) took 1271 ms. Trace:
I20260926 01:55:46.188159 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.189831 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:46.195585 9341 tablet_bootstrap.cc:492] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:46.196815 8533 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a] alter complete (version 4)
W20260926 01:55:46.205616 8532 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 22) took 1906 ms (client timeout 1000 ms). Trace:
I20260926 01:55:46.209892 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.215917 9482 raft_consensus.cc:2749] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:55:46.217864 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_cdcfa57f52a747aaa93bb04410df7061"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:46.219684 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_8c9193cf386d4eca9776e2ceb2c7dc11 [id=2af856781c904c77a11ca9f4966ca44b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 04e79a2e52fb40619b2bc99396abda5b consensus state has no leader
W20260926 01:55:46.220155 8527 catalog_manager.cc:5136] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): alter failed for tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
W20260926 01:55:46.207598 8531 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 32) took 1415 ms (client timeout 1000 ms). Trace:
W20260926 01:55:46.221189 8531 rpcz_store.cc:269] 0926 01:55:44.792601 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:44.792792 (+ 191us) service_pool.cc:224] Handling call
0926 01:55:44.841267 (+ 48475us) spinlock_profiling.cc:243] Waited 46.7 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:46.207522 (+1366255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9421,"reported_tablets":5,"spinlock_wait_cycles":107441536}
I20260926 01:55:46.222791 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:46.225792 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:46.226609 9341 tablet_bootstrap.cc:654] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.226842 9424 consensus_queue.cc:1048] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:55:46.229532 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a36d6777b8c442d99c2cd88d9110efd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:46.184526 8536 rpcz_store.cc:276] 0926 01:55:44.912618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:44.912752 (+ 134us) service_pool.cc:224] Handling call
0926 01:55:44.914114 (+ 1362us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:44.914688 (+ 574us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:44.914818 (+ 130us) catalog_manager.cc:2863] Locking tablets
0926 01:55:45.355366 (+440548us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:45.558536 (+203170us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:45.558788 (+ 252us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:46.182288 (+623500us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:46.184118 (+ 1830us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:45.451432 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:45.451814 (+ 382us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:45.451836 (+ 22us) write_op.cc:435] Acquired schema lock
0926 01:55:45.451850 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:45.487699 (+ 35849us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:45.487749 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:45.487778 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:45.488087 (+ 309us) tablet.cc:701] Row locks acquired
0926 01:55:45.488105 (+ 18us) write_op.cc:260] PREPARE: finished
0926 01:55:45.488235 (+ 130us) write_op.cc:270] Start()
0926 01:55:45.488337 (+ 102us) write_op.cc:276] Timestamp: P: 1790387745487654 usec, L: 0
0926 01:55:45.488355 (+ 18us) op_driver.cc:348] REPLICATION: starting
0926 01:55:45.488881 (+ 526us) log.cc:844] Serialized 727 byte log entry
0926 01:55:45.523806 (+ 34925us) op_driver.cc:464] REPLICATION: finished
0926 01:55:45.526018 (+ 2212us) write_op.cc:301] APPLY: starting
0926 01:55:45.526069 (+ 51us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:45.526176 (+ 107us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:45.526256 (+ 80us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:45.548262 (+ 22006us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:45.548439 (+ 177us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:45.548482 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:55:45.548695 (+ 213us) log.cc:844] Serialized 28 byte log entry
0926 01:55:45.548868 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:45.548946 (+ 78us) write_op.cc:454] Released schema lock
0926 01:55:45.548982 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:45.559011 (+ 10029us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:45.559040 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2069,"apply.run_cpu_time_us":1524,"apply.run_wall_time_us":32938,"num_ops":2,"prepare.queue_time_us":94069,"prepare.run_cpu_time_us":2353,"prepare.run_wall_time_us":71373,"raft.queue_time_us":847,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":35367}]]}
I20260926 01:55:46.184212 9339 consensus_queue.cc:260] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.248349 9516 ts_tablet_manager.cc:1929] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260926 01:55:46.249928 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:46.250761 9482 raft_consensus.cc:493] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.251224 9482 raft_consensus.cc:515] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:55:46.252094 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet dda6b1627dd846f08d717f154c3e23d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1171186cc294417db439bedf0ca4a6a2) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:46.252511 9516 log.cc:1199] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/03b827807def43b39b91d61a6dfd9386
I20260926 01:55:46.253898 9516 ts_tablet_manager.cc:1950] T 03b827807def43b39b91d61a6dfd9386 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:46.255800 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:46.255702 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d10be686d9045efaf384d326b356e0e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:46.212396 8532 rpcz_store.cc:269] 0926 01:55:44.298975 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:44.568049 (+269074us) service_pool.cc:224] Handling call
0926 01:55:44.837971 (+269922us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:46.205525 (+1367554us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6239,"reported_tablets":4,"spinlock_wait_cycles":614395264}
I20260926 01:55:46.257077 9516 tablet_replica.cc:333] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:46.257346 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:55:46.257894 9516 raft_consensus.cc:2243] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 01:55:46.259766 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdcfa57f52a747aaa93bb04410df7061 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:46.261804 8533 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 32) took 1057 ms (client timeout 1000 ms). Trace:
I20260926 01:55:46.254266 9482 leader_election.cc:290] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:46.265190 9339 ts_tablet_manager.cc:1434] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.094s user 0.005s sys 0.001s
I20260926 01:55:46.268522 9341 tablet_bootstrap.cc:492] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:46.268935 9341 ts_tablet_manager.cc:1403] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.005s
I20260926 01:55:46.271948 9072 raft_consensus.cc:3060] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:46.284363 9341 raft_consensus.cc:359] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.285034 9341 raft_consensus.cc:385] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.285288 9341 raft_consensus.cc:740] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.258365 9516 raft_consensus.cc:2272] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.289765 9516 ts_tablet_manager.cc:1916] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:46.289963 9339 tablet_bootstrap.cc:492] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:46.289615 9424 consensus_queue.cc:1048] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:46.291450 9515 ts_tablet_manager.cc:1929] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:46.291910 9515 log.cc:1199] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/1618eb529bcc4947b4c0e1e5504af71a
I20260926 01:55:46.304874 9424 consensus_queue.cc:1048] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:55:46.310621 8539 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e53c9eb8a6d648b2a9359f02b7f8764b has no consensus state
I20260926 01:55:46.312000 9515 ts_tablet_manager.cc:1950] T 1618eb529bcc4947b4c0e1e5504af71a P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:46.314843 9424 consensus_queue.cc:1048] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:46.316522 9047 tablet_service.cc:1558] Processing DeleteTablet for tablet 03b827807def43b39b91d61a6dfd9386 with delete_type TABLET_DATA_DELETED (replaced by tablet 70ef5265eb6244aa96c8195aa30cdfb0) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:46.262422 8533 rpcz_store.cc:269] 0926 01:55:45.204599 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:45.206346 (+ 1747us) service_pool.cc:224] Handling call
0926 01:55:46.261685 (+1055339us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:45.650279 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:45.650748 (+ 469us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:45.650776 (+ 28us) write_op.cc:435] Acquired schema lock
0926 01:55:45.650800 (+ 24us) tablet.cc:662] Decoding operations
0926 01:55:45.651079 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:45.651119 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:45.651141 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:45.651297 (+ 156us) tablet.cc:701] Row locks acquired
0926 01:55:45.651311 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:45.651448 (+ 137us) write_op.cc:270] Start()
0926 01:55:45.651535 (+ 87us) write_op.cc:276] Timestamp: P: 1790387745650854 usec, L: 0
0926 01:55:45.651552 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:55:45.652102 (+ 550us) log.cc:844] Serialized 529 byte log entry
0926 01:55:45.743298 (+ 91196us) op_driver.cc:464] REPLICATION: finished
0926 01:55:45.746282 (+ 2984us) write_op.cc:301] APPLY: starting
0926 01:55:45.746330 (+ 48us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:45.746422 (+ 92us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:45.755326 (+ 8904us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:45.755582 (+ 256us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:45.755725 (+ 143us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:45.755771 (+ 46us) write_op.cc:312] APPLY: finished
0926 01:55:45.755964 (+ 193us) log.cc:844] Serialized 22 byte log entry
0926 01:55:45.756127 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:45.756191 (+ 64us) write_op.cc:454] Released schema lock
0926 01:55:45.756233 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:45.762496 (+ 6263us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:45.762528 (+ 32us) write_op.cc:454] Released schema lock
Related trace 'op':
0926 01:55:45.950027 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:45.950412 (+ 385us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:45.950437 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:55:45.954499 (+ 4062us) tablet.cc:662] Decoding operations
0926 01:55:45.954836 (+ 337us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:45.954881 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:45.954905 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:45.955058 (+ 153us) tablet.cc:701] Row locks acquired
0926 01:55:45.955072 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:45.955197 (+ 125us) write_op.cc:270] Start()
0926 01:55:45.955288 (+ 91us) write_op.cc:276] Timestamp: P: 1790387745955134 usec, L: 0
0926 01:55:45.955305 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:55:45.955845 (+ 540us) log.cc:844] Serialized 355 byte log entry
0926 01:55:46.098649 (+142804us) op_driver.cc:464] REPLICATION: finished
0926 01:55:46.099153 (+ 504us) write_op.cc:301] APPLY: starting
0926 01:55:46.099223 (+ 70us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:46.099341 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:46.099441 (+ 100us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:46.099609 (+ 168us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:46.099737 (+ 128us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:46.099780 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:55:46.099976 (+ 196us) log.cc:844] Serialized 22 byte log entry
0926 01:55:46.100139 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:46.100204 (+ 65us) write_op.cc:454] Released schema lock
0926 01:55:46.100240 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:46.100488 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:46.100520 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":2854,"apply.run_cpu_time_us":1491,"apply.run_wall_time_us":16153,"num_ops":1,"prepare.queue_time_us":73377,"prepare.run_cpu_time_us":2231,"prepare.run_wall_time_us":24866,"raft.queue_time_us":1767,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":91645}],["op",{"apply.queue_time_us":323,"apply.run_cpu_time_us":1303,"apply.run_wall_time_us":1302,"num_ops":1,"prepare.queue_time_us":190424,"prepare.run_cpu_time_us":2231,"prepare.run_wall_time_us":10583,"raft.queue_time_us":165,"raft.run_cpu_time_us":731,"raft.run_wall_time_us":731,"replication_time_us":143233}]]}
I20260926 01:55:46.316208 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 1618eb529bcc4947b4c0e1e5504af71a (table default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]) successfully deleted
I20260926 01:55:46.328368 9499 raft_consensus.cc:493] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.328847 9499 raft_consensus.cc:515] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.330835 9499 leader_election.cc:290] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:46.332183 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf8818e65d74a32ba21dcb8a9a56a62" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:46.336498 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet dccbfbc9fb5947f9b85ca8560e205b08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dccbfbc9fb5947f9b85ca8560e205b08 already in progress: creating tablet
W20260926 01:55:46.337852 8843 leader_election.cc:343] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.344485 9550 consensus_queue.cc:1048] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:46.345034 9515 tablet_replica.cc:333] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:46.292402 9341 consensus_queue.cc:260] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.351277 9515 raft_consensus.cc:2243] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.353863 9341 ts_tablet_manager.cc:1434] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.085s user 0.005s sys 0.002s
I20260926 01:55:46.354780 9515 raft_consensus.cc:2272] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.352708 9339 tablet_bootstrap.cc:654] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.366110 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_523e24048b654f60bf78f4f9821423c6" } new_table_name: "default.Table_df0e68c2c9ea42e18e91228cbf77f457" modify_external_catalogs: true
I20260926 01:55:46.308202 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d10be686d9045efaf384d326b356e0e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:46.334824 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf8818e65d74a32ba21dcb8a9a56a62" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:46.372104 9217 raft_consensus.cc:2468] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:46.373598 9220 raft_consensus.cc:2393] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:55:46.373476 8843 leader_election.cc:304] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:46.374782 9550 raft_consensus.cc:2804] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:46.375126 9550 raft_consensus.cc:493] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:46.375028 9006 leader_election.cc:304] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:46.375423 9550 raft_consensus.cc:3060] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:46.378723 9341 tablet_bootstrap.cc:492] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:46.379166 9515 ts_tablet_manager.cc:1916] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:46.380607 8843 consensus_peers.cc:597] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:46.405076 9550 raft_consensus.cc:515] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.413017 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8fa5adfde54e424d935ea9df94631389" } modify_external_catalogs: true
I20260926 01:55:46.413782 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8fa5adfde54e424d935ea9df94631389" } modify_external_catalogs: true
I20260926 01:55:46.416428 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf8818e65d74a32ba21dcb8a9a56a62" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:55:46.418043 8843 leader_election.cc:343] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.419818 9341 tablet_bootstrap.cc:654] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.420312 9550 leader_election.cc:290] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:46.420428 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbf8818e65d74a32ba21dcb8a9a56a62" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:46.421010 9219 raft_consensus.cc:3060] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:46.421715 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 9d9cb4de16da44e1b90ac79e9e32cb88 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:46.423740 9339 tablet_bootstrap.cc:492] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:46.426818 9339 ts_tablet_manager.cc:1403] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.137s user 0.011s sys 0.002s
I20260926 01:55:46.432636 9482 raft_consensus.cc:2749] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:46.411116 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_1ab250b0a79b4325b2c79e44ef3ff928"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.439096 9219 raft_consensus.cc:2468] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
W20260926 01:55:46.444651 8843 consensus_peers.cc:597] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:46.448877 8532 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8e5e15ce8e58425697a7f013c09c0c72" } modify_external_catalogs: true
I20260926 01:55:46.449529 8532 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8e5e15ce8e58425697a7f013c09c0c72" } modify_external_catalogs: true
I20260926 01:55:46.450726 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet a85e47c1bf9f4e96958aa2de41039180 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:46.447173 9523 raft_consensus.cc:493] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.454694 9523 raft_consensus.cc:515] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.456712 9523 leader_election.cc:290] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:46.458384 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659eca3c97144aaacd41e747aecd96d" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:46.460361 8843 consensus_peers.cc:597] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:46.461244 9550 consensus_queue.cc:1048] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:55:46.462471 9155 leader_election.cc:343] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.464639 8843 leader_election.cc:304] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:46.465940 9550 raft_consensus.cc:2804] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:46.470614 9550 raft_consensus.cc:697] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:46.471421 9550 consensus_queue.cc:237] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.473255 9339 raft_consensus.cc:359] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.473951 9339 raft_consensus.cc:385] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.474220 9339 raft_consensus.cc:740] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.477541 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659eca3c97144aaacd41e747aecd96d" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:46.454171 9516 ts_tablet_manager.cc:1929] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260926 01:55:46.479302 9152 leader_election.cc:343] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.479303 9516 log.cc:1199] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/9d9cb4de16da44e1b90ac79e9e32cb88
I20260926 01:55:46.481003 9516 ts_tablet_manager.cc:1950] T 9d9cb4de16da44e1b90ac79e9e32cb88 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:46.479632 9339 consensus_queue.cc:260] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.489025 9516 tablet_replica.cc:333] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:46.491552 9516 raft_consensus.cc:2243] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.496758 9341 tablet_bootstrap.cc:492] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:46.497251 9341 ts_tablet_manager.cc:1403] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.005s
I20260926 01:55:46.498323 9501 consensus_queue.cc:1048] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:46.500022 9341 raft_consensus.cc:359] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.500864 9341 raft_consensus.cc:385] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.501152 9341 raft_consensus.cc:740] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.501901 9341 consensus_queue.cc:260] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.503196 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_87adaea76871454aa101fbf9112bc76a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.504292 9341 ts_tablet_manager.cc:1434] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:55:46.505165 9341 tablet_bootstrap.cc:492] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:46.506778 9516 raft_consensus.cc:2272] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.479772 9152 leader_election.cc:304] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:46.526961 9339 ts_tablet_manager.cc:1434] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.099s user 0.000s sys 0.010s
I20260926 01:55:46.531075 9523 raft_consensus.cc:2749] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:55:46.532042 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 52789.9us
I20260926 01:55:46.532996 9516 ts_tablet_manager.cc:1916] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:46.538895 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87adaea76871454aa101fbf9112bc76a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:46.484882 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 9d9cb4de16da44e1b90ac79e9e32cb88 (table default.Table_d25a39f7e9fa4682af9a020d33a580ad [id=67e50edec70245399c670c1c6aa26688]) successfully deleted
W20260926 01:55:46.542295 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 57456.3us
I20260926 01:55:46.544317 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 03b827807def43b39b91d61a6dfd9386 with delete_type TABLET_DATA_DELETED (replaced by tablet 70ef5265eb6244aa96c8195aa30cdfb0) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.546093 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.547188 9339 tablet_bootstrap.cc:492] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:46.547799 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a with delete_type TABLET_DATA_DELETED (replaced by tablet a09351c3db384b4383ca5ff0eaca982c) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.549547 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:46.550628 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23d2805b41d645d3bb905f26a380f383 has no consensus state
I20260926 01:55:46.551394 9341 tablet_bootstrap.cc:654] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.553053 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_dd6c9b79a6a34f00818d8af57fd472bb" } new_table_name: "default.Table_92899e57b6c14f62b2a999bb2487f558" modify_external_catalogs: true
I20260926 01:55:46.555986 9340 tablet_bootstrap.cc:492] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:46.556471 9340 ts_tablet_manager.cc:1403] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.566s user 0.011s sys 0.010s
W20260926 01:55:46.568197 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ab250b0a79b4325b2c79e44ef3ff928 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:46.572674 8540 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 23) took 1273 ms (client timeout 1000 ms). Trace:
W20260926 01:55:46.572985 8540 rpcz_store.cc:269] 0926 01:55:45.299388 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:45.337171 (+ 37783us) service_pool.cc:224] Handling call
0926 01:55:46.572603 (+1235432us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":618,"reported_tablets":11}
I20260926 01:55:46.575294 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.583930 9516 ts_tablet_manager.cc:1929] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:46.586318 9516 log.cc:1199] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/099af935249a4310b29852a258661794
I20260926 01:55:46.587411 9516 ts_tablet_manager.cc:1950] T 099af935249a4310b29852a258661794 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:46.587720 9532 raft_consensus.cc:493] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.588238 9532 raft_consensus.cc:515] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.569963 9340 raft_consensus.cc:359] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.592466 9340 raft_consensus.cc:385] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.592754 9340 raft_consensus.cc:740] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.595741 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 099af935249a4310b29852a258661794 (table default.Table_e13d9ec3694c418e906e55984e87a263 [id=0a1b8452d5494c0ea17e1fca2259e171]) successfully deleted
W20260926 01:55:46.598126 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 03b827807def43b39b91d61a6dfd9386 because the tablet was not found. No further retry: Not found: Tablet not found: 03b827807def43b39b91d61a6dfd9386
I20260926 01:55:46.599148 9516 tablet_replica.cc:333] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:46.605391 9516 raft_consensus.cc:2243] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.605494 9532 leader_election.cc:290] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:46.606987 9516 raft_consensus.cc:2272] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.607014 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a85e47c1bf9f4e96958aa2de41039180" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:46.610908 9340 consensus_queue.cc:260] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.619248 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a85e47c1bf9f4e96958aa2de41039180" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:46.623114 9516 ts_tablet_manager.cc:1916] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:46.623241 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_60c3b7f86e414f22b3a31ffada33212b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:46.624866 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60c3b7f86e414f22b3a31ffada33212b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:46.631152 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39cd9130640d4b14beef42e9f1b36b99 has no consensus state
I20260926 01:55:46.636608 9515 ts_tablet_manager.cc:1929] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:46.637036 9515 log.cc:1199] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/9d9cb4de16da44e1b90ac79e9e32cb88
I20260926 01:55:46.637684 9515 ts_tablet_manager.cc:1950] T 9d9cb4de16da44e1b90ac79e9e32cb88 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:46.645838 9516 ts_tablet_manager.cc:1929] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:46.646371 9516 log.cc:1199] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/3695411d251847b4b1ab0a1eb936a55a
I20260926 01:55:46.647285 9516 ts_tablet_manager.cc:1950] T 3695411d251847b4b1ab0a1eb936a55a P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:46.652016 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 3695411d251847b4b1ab0a1eb936a55a (table default.Table_37ba6de1661d4b1f95e93ea4b3348633 [id=a2b0b9ebe53d44bd99b0eae816b717fb]) successfully deleted
W20260926 01:55:46.653970 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 1618eb529bcc4947b4c0e1e5504af71a because the tablet was not found. No further retry: Not found: Tablet not found: 1618eb529bcc4947b4c0e1e5504af71a
I20260926 01:55:46.655022 9516 tablet_replica.cc:333] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:46.655932 9516 raft_consensus.cc:2243] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.656528 9516 raft_consensus.cc:2272] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.659410 9516 ts_tablet_manager.cc:1916] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:46.661334 9003 leader_election.cc:304] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:55:46.666481 9155 consensus_peers.cc:597] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:46.667155 9532 raft_consensus.cc:2749] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:46.667318 9515 tablet_replica.cc:333] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:46.668131 9515 raft_consensus.cc:2243] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.668594 9515 raft_consensus.cc:2272] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.653369 9339 tablet_bootstrap.cc:654] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.674571 9341 tablet_bootstrap.cc:492] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:46.680940 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_a9811b3931ee40d3845a8e275978c287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.667562 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 9d9cb4de16da44e1b90ac79e9e32cb88 (table default.Table_d25a39f7e9fa4682af9a020d33a580ad [id=67e50edec70245399c670c1c6aa26688]) successfully deleted
W20260926 01:55:46.684298 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9811b3931ee40d3845a8e275978c287 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:46.694800 9515 ts_tablet_manager.cc:1916] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:46.735040 9341 ts_tablet_manager.cc:1403] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.230s user 0.006s sys 0.007s
I20260926 01:55:46.728303 9567 raft_consensus.cc:493] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.716248 9523 raft_consensus.cc:493] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.743006 9523 raft_consensus.cc:515] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.745556 9550 consensus_queue.cc:1048] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:55:46.746088 9523 leader_election.cc:290] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:46.746894 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:46.748479 9155 leader_election.cc:343] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.749152 9340 ts_tablet_manager.cc:1434] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.192s user 0.004s sys 0.004s
I20260926 01:55:46.750486 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:46.756883 9152 leader_election.cc:343] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.757588 9152 leader_election.cc:304] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:46.769744 9340 tablet_bootstrap.cc:492] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:46.771163 9523 raft_consensus.cc:2749] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:46.776789 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_e18c82d7ba2541ff8790bc644e02d7ac" } new_table_name: "default.Table_a82e6ba61ea64aaa8d10956665642633" modify_external_catalogs: true
I20260926 01:55:46.780588 9532 raft_consensus.cc:493] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.781284 9532 raft_consensus.cc:515] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.783234 9532 leader_election.cc:290] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:46.785513 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a601d850375c43679dde7a46cfe71d22" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:46.787317 9514 ts_tablet_manager.cc:1929] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260926 01:55:46.787667 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a601d850375c43679dde7a46cfe71d22" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:46.789445 9514 log.cc:1199] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/03b827807def43b39b91d61a6dfd9386
I20260926 01:55:46.792956 9003 leader_election.cc:304] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:46.793814 9532 raft_consensus.cc:2749] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:46.797959 9514 ts_tablet_manager.cc:1950] T 03b827807def43b39b91d61a6dfd9386 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:46.799463 9514 tablet_replica.cc:333] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:46.801290 9516 ts_tablet_manager.cc:1929] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:46.802701 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:55:46.801925 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 1f6d9fedc7b241b59b25d591c7dc2fea (DEFAULT_TABLE table=default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.804953 8882 tablet_service.cc:1511] Processing CreateTablet for tablet dfaca3fc02634f459083d23f3102f725 (DEFAULT_TABLE table=default.Table_214d8f2884b048ebba2f07a67d5effe8 [id=bcf837a98f554432b1f529f1c6c5c4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.806028 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfaca3fc02634f459083d23f3102f725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.806352 9514 raft_consensus.cc:2243] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:46.811084 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6d9fedc7b241b59b25d591c7dc2fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.805116 9340 tablet_bootstrap.cc:654] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:46.801131 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 1f6d9fedc7b241b59b25d591c7dc2fea (DEFAULT_TABLE table=default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.802204 9197 tablet_service.cc:1511] Processing CreateTablet for tablet dfaca3fc02634f459083d23f3102f725 (DEFAULT_TABLE table=default.Table_214d8f2884b048ebba2f07a67d5effe8 [id=bcf837a98f554432b1f529f1c6c5c4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.811125 9199 tablet_service.cc:1511] Processing CreateTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d (DEFAULT_TABLE table=default.Table_8c00cbced1a64f6c9568cedb86846d44 [id=8e48f349283d44a0a8bf7ae256529a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.815452 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6d9fedc7b241b59b25d591c7dc2fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.816340 9047 tablet_service.cc:1511] Processing CreateTablet for tablet dfaca3fc02634f459083d23f3102f725 (DEFAULT_TABLE table=default.Table_214d8f2884b048ebba2f07a67d5effe8 [id=bcf837a98f554432b1f529f1c6c5c4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.816431 8885 tablet_service.cc:1511] Processing CreateTablet for tablet 01052a5e22a74a19919bae9dd0296291 (DEFAULT_TABLE table=default.Table_9c4d9e672f6f409096edc1f1995e23e7 [id=6a3cae01c8664f3d8d4b89e8a91665f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.816883 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 1f6d9fedc7b241b59b25d591c7dc2fea (DEFAULT_TABLE table=default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.817435 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfaca3fc02634f459083d23f3102f725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.817543 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01052a5e22a74a19919bae9dd0296291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.818362 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 01052a5e22a74a19919bae9dd0296291 (DEFAULT_TABLE table=default.Table_9c4d9e672f6f409096edc1f1995e23e7 [id=6a3cae01c8664f3d8d4b89e8a91665f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.818758 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfaca3fc02634f459083d23f3102f725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.819046 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 3b6daaf274894989b90da364f5da5250 (DEFAULT_TABLE table=default.Table_efe5a3dc3242481484fedd388b7be098 [id=be02dac770c84af69e2ff172417a074a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.819821 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e0fe60b8744dbb8ae2bafefe3e834d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.819867 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d (DEFAULT_TABLE table=default.Table_8c00cbced1a64f6c9568cedb86846d44 [id=8e48f349283d44a0a8bf7ae256529a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.820996 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01052a5e22a74a19919bae9dd0296291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.809782 9200 tablet_service.cc:1511] Processing CreateTablet for tablet 01052a5e22a74a19919bae9dd0296291 (DEFAULT_TABLE table=default.Table_9c4d9e672f6f409096edc1f1995e23e7 [id=6a3cae01c8664f3d8d4b89e8a91665f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.821842 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01052a5e22a74a19919bae9dd0296291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.823483 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e0fe60b8744dbb8ae2bafefe3e834d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.823992 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6daaf274894989b90da364f5da5250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.823801 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d (DEFAULT_TABLE table=default.Table_8c00cbced1a64f6c9568cedb86846d44 [id=8e48f349283d44a0a8bf7ae256529a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.824492 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6d9fedc7b241b59b25d591c7dc2fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.824839 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32e0fe60b8744dbb8ae2bafefe3e834d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.814083 8537 catalog_manager.cc:5838] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:46.837895 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_a9811b3931ee40d3845a8e275978c287" } new_table_name: "default.Table_722ce830650d49ffa0b00028e056e1db" modify_external_catalogs: true
I20260926 01:55:46.843356 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8b359d50e9ec420e921fdce8a36cb4f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.844455 9523 raft_consensus.cc:493] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.844254 9570 tablet.cc:1751] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:46.845024 9523 raft_consensus.cc:515] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.847792 9523 leader_election.cc:290] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:46.848317 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562604137d484e83a51de2f778d58fca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:46.850369 9155 leader_election.cc:343] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.851342 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562604137d484e83a51de2f778d58fca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:46.851991 8906 raft_consensus.cc:2468] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:46.854063 9152 leader_election.cc:304] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:46.855618 9523 raft_consensus.cc:2804] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:46.856009 9523 raft_consensus.cc:493] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:46.856369 9523 raft_consensus.cc:3060] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:46.856834 9515 ts_tablet_manager.cc:1929] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:46.857295 9515 log.cc:1199] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/dda6b1627dd846f08d717f154c3e23d9
I20260926 01:55:46.857940 9515 ts_tablet_manager.cc:1950] T dda6b1627dd846f08d717f154c3e23d9 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:46.836704 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]). Expected version 1 got 0
W20260926 01:55:46.858639 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b359d50e9ec420e921fdce8a36cb4f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:46.859139 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:46.860344 8537 catalog_manager.cc:5838] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:46.862542 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 1 got 0
I20260926 01:55:46.863325 8537 catalog_manager.cc:5838] T 4411120140f3410c9f4e33e26da20d98 P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:55:46.881558 9516 log.cc:1199] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/28bd8695a6b34585bdb1bdabf1bcc654
I20260926 01:55:46.882409 9516 ts_tablet_manager.cc:1950] T 28bd8695a6b34585bdb1bdabf1bcc654 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:46.882543 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 3b6daaf274894989b90da364f5da5250 (DEFAULT_TABLE table=default.Table_efe5a3dc3242481484fedd388b7be098 [id=be02dac770c84af69e2ff172417a074a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.883697 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6daaf274894989b90da364f5da5250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.884116 9567 raft_consensus.cc:515] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.887051 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 28bd8695a6b34585bdb1bdabf1bcc654 (table default.Table_27cdeab15da2426d85cbbb43441e39d6 [id=3b732759201f4eaab4fdf76cd8a38f7e]) successfully deleted
I20260926 01:55:46.888741 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a54eb3b8cf64a478df38df9d4aef02f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:46.892046 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a54eb3b8cf64a478df38df9d4aef02f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:46.893486 9155 leader_election.cc:343] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:46.897986 8887 tablet_service.cc:1511] Processing CreateTablet for tablet 3b6daaf274894989b90da364f5da5250 (DEFAULT_TABLE table=default.Table_efe5a3dc3242481484fedd388b7be098 [id=be02dac770c84af69e2ff172417a074a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.899152 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6daaf274894989b90da364f5da5250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.910722 9514 pending_rounds.cc:70] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Trying to abort 1 pending ops.
I20260926 01:55:46.911137 9514 pending_rounds.cc:77] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428204722601984 op_type: NO_OP noop_request { }
I20260926 01:55:46.911587 9514 raft_consensus.cc:2889] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:46.912163 9514 raft_consensus.cc:2272] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.914390 9514 ts_tablet_manager.cc:1916] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:46.914479 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b (DEFAULT_TABLE table=default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.915670 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.927269 9516 tablet_replica.cc:333] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:46.934753 9339 tablet_bootstrap.cc:492] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:46.935231 9339 ts_tablet_manager.cc:1403] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.388s user 0.024s sys 0.008s
I20260926 01:55:46.937036 9516 raft_consensus.cc:2243] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.937698 9339 raft_consensus.cc:359] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.938886 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_235a08a4f73a4761935c03af88c3a393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.939894 9070 raft_consensus.cc:3060] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:46.940762 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_235a08a4f73a4761935c03af88c3a393 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:46.942909 9516 raft_consensus.cc:2272] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.949592 8534 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 75d4598596ff49378fc3536fff0ac1b1
I20260926 01:55:46.951697 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:46.954267 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 1866cd97bb024a9baecf101a6d5b1ee4
I20260926 01:55:46.957764 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:46.971947 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 (DEFAULT_TABLE table=default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.973157 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df1d4daaddc4ac4aa02aa7763ea3184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.974283 9572 raft_consensus.cc:493] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:46.974795 9572 raft_consensus.cc:515] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.975546 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet dda6b1627dd846f08d717f154c3e23d9 (table default.Table_c2f17033c99e4c2d9fa8181e746f5359 [id=4560c208b2344ba5913dbd1d30a6d09e]) successfully deleted
W20260926 01:55:46.975880 8540 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c36ea09cf15647c0a344c9ba9b289d61 has no consensus state
I20260926 01:55:46.976439 9515 tablet_replica.cc:333] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:46.977245 9515 raft_consensus.cc:2243] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:46.978178 9515 raft_consensus.cc:2272] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:46.938315 9339 raft_consensus.cc:385] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:46.978631 9339 raft_consensus.cc:740] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:46.978971 8540 master_service.cc:960] ReplaceTablet: received request to replace tablet 8a54eb3b8cf64a478df38df9d4aef02f from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:46.979369 9195 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.979296 9339 consensus_queue.cc:260] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:46.981164 9339 ts_tablet_manager.cc:1434] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.046s user 0.004s sys 0.000s
I20260926 01:55:46.982189 9339 tablet_bootstrap.cc:492] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:46.982952 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b (DEFAULT_TABLE table=default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.984174 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:46.984318 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_88a0ad1233b9421a8cdfc42b5b156240"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.994306 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:46.995215 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_2b020c7bf3ca40f5932fadfe0dc566d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:46.995721 9567 leader_election.cc:290] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:46.997284 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b020c7bf3ca40f5932fadfe0dc566d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:47.001693 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88a0ad1233b9421a8cdfc42b5b156240 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:46.939400 9340 tablet_bootstrap.cc:492] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:47.003741 9340 ts_tablet_manager.cc:1403] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.234s user 0.009s sys 0.005s
I20260926 01:55:47.007499 9341 raft_consensus.cc:359] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.008183 9341 raft_consensus.cc:385] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.008433 9341 raft_consensus.cc:740] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.009093 9341 consensus_queue.cc:260] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:46.943677 9569 tablet.cc:1751] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:46.954356 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:46.967216 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:46.972218 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b (DEFAULT_TABLE table=default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:46.978674 8879 tablet_service.cc:1511] Processing CreateTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 (DEFAULT_TABLE table=default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.012243 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 (DEFAULT_TABLE table=default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.013463 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df1d4daaddc4ac4aa02aa7763ea3184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.015616 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 (DEFAULT_TABLE table=default.Table_8fa5adfde54e424d935ea9df94631389 [id=d706c478a4b54165a5e8943317180616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.018592 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.011653 8536 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_d7957d605b514c30a979aa567bec903e [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 1 got 0
I20260926 01:55:47.021090 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 (DEFAULT_TABLE table=default.Table_8fa5adfde54e424d935ea9df94631389 [id=d706c478a4b54165a5e8943317180616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.021524 8536 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]). Expected version 1 got 0
I20260926 01:55:47.022290 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75d4598596ff49378fc3536fff0ac1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.022768 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df1d4daaddc4ac4aa02aa7763ea3184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.022639 8536 catalog_manager.cc:5838] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:47.024333 8536 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:47.035211 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75d4598596ff49378fc3536fff0ac1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.035100 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:47.006160 9340 raft_consensus.cc:359] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.036914 9340 raft_consensus.cc:385] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.037194 9340 raft_consensus.cc:740] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.037845 9340 consensus_queue.cc:260] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.040395 9574 tablet.cc:1751] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:47.041220 8536 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:47.041998 8536 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 1 got 0
I20260926 01:55:47.023288 9515 ts_tablet_manager.cc:1916] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:47.051028 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 (DEFAULT_TABLE table=default.Table_8fa5adfde54e424d935ea9df94631389 [id=d706c478a4b54165a5e8943317180616]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:47.047973 8537 catalog_manager.cc:6038] Failed to send AlterTable RPC for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 8dfff96cc29145e7a72213bb5ecca1ae consensus state has no leader
I20260926 01:55:47.052297 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75d4598596ff49378fc3536fff0ac1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.055503 8536 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_8c9193cf386d4eca9776e2ceb2c7dc11 [id=2af856781c904c77a11ca9f4966ca44b]). Expected version 2 got 0
I20260926 01:55:47.067476 9199 tablet_service.cc:1511] Processing CreateTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f (DEFAULT_TABLE table=default.Table_48399c44f8254aaeaa623db43553e1ca [id=eb2d6f7b9c9a48c49b34d0617267301e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.068737 9199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3bb1643da054583b97efd2e1a9e9c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.069339 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 23d2805b41d645d3bb905f26a380f383 (DEFAULT_TABLE table=default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.070600 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d2805b41d645d3bb905f26a380f383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.072649 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04e79a2e52fb40619b2bc99396abda5b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:47.074147 9155 leader_election.cc:343] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:47.076037 9572 leader_election.cc:290] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:47.078039 9523 raft_consensus.cc:515] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.080219 9046 tablet_service.cc:1511] Processing CreateTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f (DEFAULT_TABLE table=default.Table_48399c44f8254aaeaa623db43553e1ca [id=eb2d6f7b9c9a48c49b34d0617267301e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.081458 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3bb1643da054583b97efd2e1a9e9c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.087460 9339 tablet_bootstrap.cc:654] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:47.090699 8537 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 33) took 998 ms (client timeout 1000 ms). Trace:
W20260926 01:55:47.091147 8537 rpcz_store.cc:269] 0926 01:55:46.092790 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.161523 (+ 68733us) service_pool.cc:224] Handling call
0926 01:55:47.090618 (+929095us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:46.868040 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:46.868445 (+ 405us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:46.868468 (+ 23us) write_op.cc:435] Acquired schema lock
0926 01:55:46.868481 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:46.868915 (+ 434us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:46.868955 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:46.868979 (+ 24us) tablet.cc:685] Acquiring locks for 3 operations
0926 01:55:46.869238 (+ 259us) tablet.cc:701] Row locks acquired
0926 01:55:46.869253 (+ 15us) write_op.cc:260] PREPARE: finished
0926 01:55:46.869373 (+ 120us) write_op.cc:270] Start()
0926 01:55:46.869458 (+ 85us) write_op.cc:276] Timestamp: P: 1790387746869324 usec, L: 0
0926 01:55:46.869474 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:46.869982 (+ 508us) log.cc:844] Serialized 1284 byte log entry
0926 01:55:46.963933 (+ 93951us) op_driver.cc:464] REPLICATION: finished
0926 01:55:46.968897 (+ 4964us) write_op.cc:301] APPLY: starting
0926 01:55:46.968955 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:46.969095 (+ 140us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:46.969199 (+ 104us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:46.969679 (+ 480us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:46.969815 (+ 136us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0926 01:55:46.969863 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:55:46.970051 (+ 188us) log.cc:844] Serialized 35 byte log entry
0926 01:55:46.970170 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:46.970246 (+ 76us) write_op.cc:454] Released schema lock
0926 01:55:47.000762 (+ 30516us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:47.029780 (+ 29018us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:47.029808 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":3556,"apply.run_cpu_time_us":2386,"apply.run_wall_time_us":32096,"num_ops":3,"prepare.queue_time_us":1646,"prepare.run_cpu_time_us":2324,"prepare.run_wall_time_us":161711,"raft.queue_time_us":2392,"raft.run_cpu_time_us":783,"raft.run_wall_time_us":1841,"replication_time_us":94175}]]}
I20260926 01:55:47.107510 9340 ts_tablet_manager.cc:1434] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.102s user 0.005s sys 0.001s
W20260926 01:55:47.115499 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 80814.2us
I20260926 01:55:47.124099 9053 tablet_service.cc:1511] Processing CreateTablet for tablet a09351c3db384b4383ca5ff0eaca982c (DEFAULT_TABLE table=default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.125409 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09351c3db384b4383ca5ff0eaca982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.134987 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 03b827807def43b39b91d61a6dfd9386 with delete_type TABLET_DATA_DELETED (replaced by tablet 70ef5265eb6244aa96c8195aa30cdfb0) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:47.136729 8883 tablet_service.cc:1511] Processing CreateTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f (DEFAULT_TABLE table=default.Table_48399c44f8254aaeaa623db43553e1ca [id=eb2d6f7b9c9a48c49b34d0617267301e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.137851 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3bb1643da054583b97efd2e1a9e9c2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:47.143837 8843 consensus_peers.cc:597] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 099af935249a4310b29852a258661794. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:47.143716 8886 tablet_service.cc:1511] Processing CreateTablet for tablet 23d2805b41d645d3bb905f26a380f383 (DEFAULT_TABLE table=default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.145074 8878 tablet_service.cc:1511] Processing CreateTablet for tablet a09351c3db384b4383ca5ff0eaca982c (DEFAULT_TABLE table=default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.145519 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d2805b41d645d3bb905f26a380f383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.146298 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09351c3db384b4383ca5ff0eaca982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.165164 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562604137d484e83a51de2f778d58fca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:55:47.166752 9155 leader_election.cc:343] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:47.167742 9523 leader_election.cc:290] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:47.169005 9340 tablet_bootstrap.cc:492] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:47.082499 9516 ts_tablet_manager.cc:1916] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:47.097733 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 23d2805b41d645d3bb905f26a380f383 (DEFAULT_TABLE table=default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.175021 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d2805b41d645d3bb905f26a380f383. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:47.099889 9152 leader_election.cc:343] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:47.190479 9340 tablet_bootstrap.cc:654] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:47.192472 9550 consensus_queue.cc:1048] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:47.203972 8540 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 8a54eb3b8cf64a478df38df9d4aef02f
I20260926 01:55:47.205550 8540 catalog_manager.cc:6934] ReplaceTablet: tablet 8a54eb3b8cf64a478df38df9d4aef02f deleted and replaced by tablet 3571967749844fd7859cf179a93cf403
I20260926 01:55:47.206192 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f with delete_type TABLET_DATA_DELETED (replaced by tablet 3571967749844fd7859cf179a93cf403) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:47.135388 9341 ts_tablet_manager.cc:1434] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.395s user 0.002s sys 0.004s
I20260926 01:55:47.208220 9341 tablet_bootstrap.cc:492] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:47.210952 8535 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_48399c44f8254aaeaa623db43553e1ca" } modify_external_catalogs: true
I20260926 01:55:47.211145 9051 tablet_service.cc:1511] Processing CreateTablet for tablet e53c9eb8a6d648b2a9359f02b7f8764b (DEFAULT_TABLE table=default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.211503 8535 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_48399c44f8254aaeaa623db43553e1ca" } modify_external_catalogs: true
I20260926 01:55:47.212355 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53c9eb8a6d648b2a9359f02b7f8764b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.216073 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f with delete_type TABLET_DATA_DELETED (replaced by tablet 3571967749844fd7859cf179a93cf403) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:47.216058 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_60c3b7f86e414f22b3a31ffada33212b" } new_table_name: "default.Table_1871a77d01894075a74872c5af9622ec" modify_external_catalogs: true
I20260926 01:55:47.220013 8537 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_2a36d6777b8c442d99c2cd88d9110efd" } modify_external_catalogs: true
I20260926 01:55:47.220861 8537 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_2a36d6777b8c442d99c2cd88d9110efd" } modify_external_catalogs: true
W20260926 01:55:47.231060 8525 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8dfff96cc29145e7a72213bb5ecca1ae consensus state has no leader
I20260926 01:55:47.247064 9195 tablet_service.cc:1511] Processing CreateTablet for tablet a09351c3db384b4383ca5ff0eaca982c (DEFAULT_TABLE table=default.Table_e18c82d7ba2541ff8790bc644e02d7ac [id=008c87e2e12f48519efdacc69466442a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.252954 8884 tablet_service.cc:1511] Processing CreateTablet for tablet e53c9eb8a6d648b2a9359f02b7f8764b (DEFAULT_TABLE table=default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.253511 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f with delete_type TABLET_DATA_DELETED (replaced by tablet 3571967749844fd7859cf179a93cf403) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:47.259184 9340 tablet_bootstrap.cc:492] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:47.259649 9340 ts_tablet_manager.cc:1403] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.091s user 0.020s sys 0.009s
I20260926 01:55:47.262189 9340 raft_consensus.cc:359] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.264088 9340 raft_consensus.cc:385] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.264377 9340 raft_consensus.cc:740] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.265003 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53c9eb8a6d648b2a9359f02b7f8764b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.265079 9340 consensus_queue.cc:260] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.268836 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09351c3db384b4383ca5ff0eaca982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.278393 9340 ts_tablet_manager.cc:1434] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260926 01:55:47.278286 9194 tablet_service.cc:1511] Processing CreateTablet for tablet e53c9eb8a6d648b2a9359f02b7f8764b (DEFAULT_TABLE table=default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:47.237211 9152 leader_election.cc:304] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
W20260926 01:55:47.279681 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 179827us
I20260926 01:55:47.281745 9523 raft_consensus.cc:2749] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:47.284308 9340 tablet_bootstrap.cc:492] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:47.285370 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04e79a2e52fb40619b2bc99396abda5b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:47.285979 8907 raft_consensus.cc:2468] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:47.300861 9072 raft_consensus.cc:3060] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.312567 9339 tablet_bootstrap.cc:492] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:47.313221 9339 ts_tablet_manager.cc:1403] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.331s user 0.016s sys 0.006s
I20260926 01:55:47.321941 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]). Expected version 1 got 0
I20260926 01:55:47.327426 9340 tablet_bootstrap.cc:654] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:47.323230 9515 ts_tablet_manager.cc:1929] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260926 01:55:47.328660 9515 log.cc:1199] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/03b827807def43b39b91d61a6dfd9386
I20260926 01:55:47.329496 9515 ts_tablet_manager.cc:1950] T 03b827807def43b39b91d61a6dfd9386 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:47.331852 9515 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 61) took 1084 ms. Trace:
I20260926 01:55:47.332257 9515 rpcz_store.cc:276] 0926 01:55:46.248884 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.316349 (+ 67465us) service_pool.cc:224] Handling call
0926 01:55:46.976271 (+659922us) ts_tablet_manager.cc:1151] Deleting tablet 03b827807def43b39b91d61a6dfd9386
0926 01:55:46.976321 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:47.172444 (+196123us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:47.323076 (+150632us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:47.331782 (+ 8706us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":659266}
I20260926 01:55:47.334160 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 03b827807def43b39b91d61a6dfd9386 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
W20260926 01:55:47.340342 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a09351c3db384b4383ca5ff0eaca982c consensus state has no leader
W20260926 01:55:47.340329 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 75d4598596ff49378fc3536fff0ac1b1 because the tablet was not found. No further retry: Not found: Tablet not found: 75d4598596ff49378fc3536fff0ac1b1
I20260926 01:55:47.361075 9340 tablet_bootstrap.cc:492] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:47.361555 9340 ts_tablet_manager.cc:1403] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.000s
I20260926 01:55:47.363617 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_7f0d970c20fe4172b726678f7ddcb661"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:47.364140 9340 raft_consensus.cc:359] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.364364 9339 raft_consensus.cc:359] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.364730 9340 raft_consensus.cc:385] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.364990 9339 raft_consensus.cc:385] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.364967 9340 raft_consensus.cc:740] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.365235 9339 raft_consensus.cc:740] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.365725 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet dda6b1627dd846f08d717f154c3e23d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1171186cc294417db439bedf0ca4a6a2) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:47.365552 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f0d970c20fe4172b726678f7ddcb661 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:47.370738 9339 consensus_queue.cc:260] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.373101 9339 ts_tablet_manager.cc:1434] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20260926 01:55:47.374004 9339 tablet_bootstrap.cc:492] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:55:47.379087 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 1866cd97bb024a9baecf101a6d5b1ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1866cd97bb024a9baecf101a6d5b1ee4 already in progress: creating tablet
I20260926 01:55:47.378800 9340 consensus_queue.cc:260] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.383338 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:47.383869 9341 tablet_bootstrap.cc:654] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:47.388657 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 03b827807def43b39b91d61a6dfd9386 with delete_type TABLET_DATA_DELETED (replaced by tablet 70ef5265eb6244aa96c8195aa30cdfb0) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:47.390017 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 8a54eb3b8cf64a478df38df9d4aef02f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a54eb3b8cf64a478df38df9d4aef02f already in progress: creating tablet
W20260926 01:55:47.391798 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet dda6b1627dd846f08d717f154c3e23d9 because the tablet was not found. No further retry: Not found: Tablet not found: dda6b1627dd846f08d717f154c3e23d9
W20260926 01:55:47.393388 8538 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 33) took 934 ms (client timeout 1000 ms). Trace:
I20260926 01:55:47.394548 9340 ts_tablet_manager.cc:1434] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.033s user 0.007s sys 0.001s
I20260926 01:55:47.394493 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 1618eb529bcc4947b4c0e1e5504af71a with delete_type TABLET_DATA_DELETED (replaced by tablet a09351c3db384b4383ca5ff0eaca982c) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:47.394753 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:47.395213 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:47.395237 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:47.423306 9516 ts_tablet_manager.cc:1929] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:47.401398 9515 tablet_replica.cc:333] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:47.422385 9514 ts_tablet_manager.cc:1929] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:47.430220 9515 raft_consensus.cc:2243] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:47.430672 9514 log.cc:1199] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/9d9cb4de16da44e1b90ac79e9e32cb88
I20260926 01:55:47.430846 9515 raft_consensus.cc:2272] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:47.433980 9515 ts_tablet_manager.cc:1916] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:47.420027 9532 raft_consensus.cc:493] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:47.434893 9532 raft_consensus.cc:515] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:47.399945 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 because the tablet was not found. No further retry: Not found: Tablet not found: 9d9cb4de16da44e1b90ac79e9e32cb88
I20260926 01:55:47.435478 8534 master_service.cc:960] ReplaceTablet: received request to replace tablet 32e0fe60b8744dbb8ae2bafefe3e834d from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:47.435546 9514 ts_tablet_manager.cc:1950] T 9d9cb4de16da44e1b90ac79e9e32cb88 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:47.406517 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_851c22c2b5c04d179232ac610720efd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:47.436012 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 03b827807def43b39b91d61a6dfd9386 because the tablet was not found. No further retry: Not found: Tablet not found: 03b827807def43b39b91d61a6dfd9386
W20260926 01:55:47.436578 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 1618eb529bcc4947b4c0e1e5504af71a because the tablet was not found. No further retry: Not found: Tablet not found: 1618eb529bcc4947b4c0e1e5504af71a
I20260926 01:55:47.436961 9532 leader_election.cc:290] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:47.436971 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_851c22c2b5c04d179232ac610720efd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:47.437096 8538 rpcz_store.cc:269] 0926 01:55:46.458699 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.486605 (+ 27906us) service_pool.cc:224] Handling call
0926 01:55:47.393323 (+906718us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":9}
I20260926 01:55:47.437999 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 67) took 1248 ms. Trace:
I20260926 01:55:47.439599 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3695411d251847b4b1ab0a1eb936a55a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:47.429068 9339 tablet_bootstrap.cc:654] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:47.450925 9340 tablet_bootstrap.cc:492] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:47.438380 9514 rpcz_store.cc:276] 0926 01:55:46.189525 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.189678 (+ 153us) service_pool.cc:224] Handling call
0926 01:55:46.799293 (+609615us) ts_tablet_manager.cc:1151] Deleting tablet 9d9cb4de16da44e1b90ac79e9e32cb88
0926 01:55:46.799334 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:47.162284 (+362950us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:47.422203 (+259919us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:47.437917 (+ 15714us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99230,"tablet-delete.queue_time_us":609020}
I20260926 01:55:47.459218 9514 tablet_replica.cc:333] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:47.460096 9514 raft_consensus.cc:2243] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
W20260926 01:55:47.435145 8536 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 24) took 1009 ms (client timeout 1000 ms). Trace:
I20260926 01:55:47.452330 8535 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet f3bb1643da054583b97efd2e1a9e9c2f
I20260926 01:55:47.466202 9339 tablet_bootstrap.cc:492] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:47.458530 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 9d9cb4de16da44e1b90ac79e9e32cb88 (table default.Table_d25a39f7e9fa4682af9a020d33a580ad [id=67e50edec70245399c670c1c6aa26688]) successfully deleted
I20260926 01:55:47.466694 9339 ts_tablet_manager.cc:1403] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.000s
I20260926 01:55:47.463812 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_6ba9495c267245aa86a1b9184b17a7f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:47.468438 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:47.468523 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ba9495c267245aa86a1b9184b17a7f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:47.470180 9339 raft_consensus.cc:359] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.470896 9339 raft_consensus.cc:385] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.471140 9339 raft_consensus.cc:740] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.467940 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:47.472556 9514 raft_consensus.cc:2272] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:47.470129 8536 rpcz_store.cc:269] 0926 01:55:46.425860 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.426173 (+ 313us) service_pool.cc:224] Handling call
0926 01:55:47.435061 (+1008888us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:47.274435 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:47.274941 (+ 506us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:47.274968 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:55:47.274983 (+ 15us) tablet.cc:662] Decoding operations
0926 01:55:47.275274 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:47.275318 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:47.275340 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:47.275500 (+ 160us) tablet.cc:701] Row locks acquired
0926 01:55:47.275512 (+ 12us) write_op.cc:260] PREPARE: finished
0926 01:55:47.275646 (+ 134us) write_op.cc:270] Start()
0926 01:55:47.275741 (+ 95us) write_op.cc:276] Timestamp: P: 1790387747275610 usec, L: 0
0926 01:55:47.275757 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:47.276306 (+ 549us) log.cc:844] Serialized 530 byte log entry
0926 01:55:47.306064 (+ 29758us) op_driver.cc:464] REPLICATION: finished
0926 01:55:47.306480 (+ 416us) write_op.cc:301] APPLY: starting
0926 01:55:47.306538 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:47.306657 (+ 119us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:47.306772 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:47.306967 (+ 195us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:47.307101 (+ 134us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:47.307146 (+ 45us) write_op.cc:312] APPLY: finished
0926 01:55:47.307350 (+ 204us) log.cc:844] Serialized 23 byte log entry
0926 01:55:47.307506 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:47.307580 (+ 74us) write_op.cc:454] Released schema lock
0926 01:55:47.307621 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:47.372960 (+ 65339us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:47.372989 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":230,"apply.run_cpu_time_us":1359,"apply.run_wall_time_us":1358,"num_ops":1,"prepare.queue_time_us":216747,"prepare.run_cpu_time_us":2343,"prepare.run_wall_time_us":97395,"raft.queue_time_us":5552,"raft.run_cpu_time_us":829,"raft.run_wall_time_us":1741,"replication_time_us":30189}]]}
I20260926 01:55:47.474665 9514 ts_tablet_manager.cc:1916] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:47.492156 8540 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a56044249bb4aaa81903d2a8aeb008e has no consensus state
I20260926 01:55:47.492815 9339 consensus_queue.cc:260] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.475314 9340 tablet_bootstrap.cc:654] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:47.511041 9339 ts_tablet_manager.cc:1434] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260926 01:55:47.512233 9339 tablet_bootstrap.cc:492] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:47.529078 8537 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 6ba0c4e68e4a40a3bcc8fd37855ccef1
I20260926 01:55:47.531313 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562604137d484e83a51de2f778d58fca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:47.531985 8907 raft_consensus.cc:3060] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.541747 8907 raft_consensus.cc:2468] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:47.545502 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d7957d605b514c30a979aa567bec903e" } new_table_name: "default.Table_24ac8de493eb496bb7aeb552fef99c9c" modify_external_catalogs: true
I20260926 01:55:47.557161 9340 tablet_bootstrap.cc:492] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:47.557685 9340 ts_tablet_manager.cc:1403] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.107s user 0.007s sys 0.006s
I20260926 01:55:47.565285 9523 consensus_queue.cc:1048] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:55:47.565311 9516 log.cc:1199] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/9d10be686d9045efaf384d326b356e0e
I20260926 01:55:47.568288 9516 ts_tablet_manager.cc:1950] T 9d10be686d9045efaf384d326b356e0e P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:47.569504 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet a85e47c1bf9f4e96958aa2de41039180
I20260926 01:55:47.540060 9152 leader_election.cc:304] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:47.571210 8531 catalog_manager.cc:6934] ReplaceTablet: tablet a85e47c1bf9f4e96958aa2de41039180 deleted and replaced by tablet f1f020590a8b428788e2bb516a11cc45
I20260926 01:55:47.571592 8531 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 122) took 1123 ms. Trace:
I20260926 01:55:47.554909 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:47.561033 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53c9eb8a6d648b2a9359f02b7f8764b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:47.573380 9516 tablet_replica.cc:333] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:47.573632 8539 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_d7957d605b514c30a979aa567bec903e [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 1 got 0
I20260926 01:55:47.574805 9516 raft_consensus.cc:2243] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:47.575433 9516 raft_consensus.cc:2272] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:47.578550 9572 raft_consensus.cc:493] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:47.563045 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:47.579072 9572 raft_consensus.cc:515] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.579605 9523 raft_consensus.cc:2804] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:47.579918 9523 raft_consensus.cc:493] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:47.580237 9523 raft_consensus.cc:3060] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.581301 9572 leader_election.cc:290] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:47.585764 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:47.586803 9341 tablet_bootstrap.cc:492] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:47.587370 9341 ts_tablet_manager.cc:1403] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.379s user 0.013s sys 0.014s
I20260926 01:55:47.588644 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:47.590655 9341 raft_consensus.cc:359] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.591377 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:47.577941 9516 ts_tablet_manager.cc:1916] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:47.591535 8539 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:47.591840 9341 raft_consensus.cc:385] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.592013 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet a85e47c1bf9f4e96958aa2de41039180 with delete_type TABLET_DATA_DELETED (replaced by tablet f1f020590a8b428788e2bb516a11cc45) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:47.592119 9341 raft_consensus.cc:740] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.592816 9341 consensus_queue.cc:260] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.592398 8539 catalog_manager.cc:5838] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:47.594107 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet a85e47c1bf9f4e96958aa2de41039180 with delete_type TABLET_DATA_DELETED (replaced by tablet f1f020590a8b428788e2bb516a11cc45) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:47.605660 9341 ts_tablet_manager.cc:1434] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20260926 01:55:47.607790 9341 tablet_bootstrap.cc:492] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:47.571877 8531 rpcz_store.cc:276] 0926 01:55:46.448336 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.450583 (+ 2247us) service_pool.cc:224] Handling call
0926 01:55:47.571528 (+1120945us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:47.539325 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:47.539752 (+ 427us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:47.539777 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:55:47.539791 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:47.550423 (+ 10632us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:47.550502 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:47.550527 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:47.550776 (+ 249us) tablet.cc:701] Row locks acquired
0926 01:55:47.550790 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:47.550929 (+ 139us) write_op.cc:270] Start()
0926 01:55:47.551028 (+ 99us) write_op.cc:276] Timestamp: P: 1790387747550881 usec, L: 0
0926 01:55:47.551046 (+ 18us) op_driver.cc:348] REPLICATION: starting
0926 01:55:47.551625 (+ 579us) log.cc:844] Serialized 659 byte log entry
0926 01:55:47.565818 (+ 14193us) op_driver.cc:464] REPLICATION: finished
0926 01:55:47.567111 (+ 1293us) write_op.cc:301] APPLY: starting
0926 01:55:47.567167 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:47.567297 (+ 130us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:47.567409 (+ 112us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:47.567767 (+ 358us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:47.567982 (+ 215us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:47.568028 (+ 46us) write_op.cc:312] APPLY: finished
0926 01:55:47.568212 (+ 184us) log.cc:844] Serialized 29 byte log entry
0926 01:55:47.568362 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:47.568428 (+ 66us) write_op.cc:454] Released schema lock
0926 01:55:47.568458 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:47.568684 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:47.568707 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1134,"apply.run_cpu_time_us":1544,"apply.run_wall_time_us":1544,"num_ops":2,"prepare.queue_time_us":214033,"prepare.run_cpu_time_us":2425,"prepare.run_wall_time_us":24325,"raft.queue_time_us":93,"raft.run_cpu_time_us":846,"raft.run_wall_time_us":847,"replication_time_us":14678}]]}
I20260926 01:55:47.613261 9340 raft_consensus.cc:359] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.613972 9340 raft_consensus.cc:385] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.614253 9340 raft_consensus.cc:740] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.615056 9340 consensus_queue.cc:260] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.617899 9072 raft_consensus.cc:3060] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.621009 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dccbfbc9fb5947f9b85ca8560e205b08" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:47.622686 9073 raft_consensus.cc:3060] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.626922 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_0818779c2b1449eaaee8c359e22c3b86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:47.627440 9155 leader_election.cc:343] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:47.634552 9152 leader_election.cc:304] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
W20260926 01:55:47.635180 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0818779c2b1449eaaee8c359e22c3b86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:47.635464 9572 raft_consensus.cc:2804] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:55:47.625662 9006 leader_election.cc:343] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 3695411d251847b4b1ab0a1eb936a55a
I20260926 01:55:47.636965 9006 leader_election.cc:304] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:47.648096 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dccbfbc9fb5947f9b85ca8560e205b08" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:47.648813 8906 raft_consensus.cc:2468] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:47.655907 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 9d10be686d9045efaf384d326b356e0e (table default.Table_bf355929f0d5467bb45b71424eb7155b [id=2be262701c0b465f960fc3d666a78026]) successfully deleted
I20260926 01:55:47.662086 9339 tablet_bootstrap.cc:654] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:47.663069 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 1866cd97bb024a9baecf101a6d5b1ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1866cd97bb024a9baecf101a6d5b1ee4 already in progress: creating tablet
W20260926 01:55:47.667691 8843 consensus_peers.cc:597] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3695411d251847b4b1ab0a1eb936a55a. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:55:47.668522 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 75d4598596ff49378fc3536fff0ac1b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75d4598596ff49378fc3536fff0ac1b1 already in progress: creating tablet
I20260926 01:55:47.671597 9152 leader_election.cc:304] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:47.673420 9340 ts_tablet_manager.cc:1434] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.115s user 0.004s sys 0.004s
I20260926 01:55:47.675640 9515 ts_tablet_manager.cc:1929] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:47.676062 9515 log.cc:1199] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/099af935249a4310b29852a258661794
I20260926 01:55:47.688927 9515 ts_tablet_manager.cc:1950] T 099af935249a4310b29852a258661794 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:47.674674 9550 consensus_queue.cc:1048] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:47.711478 9523 raft_consensus.cc:515] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.717653 9595 raft_consensus.cc:2804] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:47.718127 9595 raft_consensus.cc:493] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:47.719077 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_87adaea76871454aa101fbf9112bc76a" } modify_external_catalogs: true
I20260926 01:55:47.719647 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_87adaea76871454aa101fbf9112bc76a" } modify_external_catalogs: true
I20260926 01:55:47.724123 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8c00cbced1a64f6c9568cedb86846d44" } new_table_name: "default.Table_17296161bc2343d1acf54e54e6e59e3b" modify_external_catalogs: true
I20260926 01:55:47.733606 8532 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_efe5a3dc3242481484fedd388b7be098" } modify_external_catalogs: true
I20260926 01:55:47.734213 8532 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_efe5a3dc3242481484fedd388b7be098" } modify_external_catalogs: true
W20260926 01:55:47.677237 8843 consensus_peers.cc:597] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d10be686d9045efaf384d326b356e0e. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:47.695595 9340 tablet_bootstrap.cc:492] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:55:47.740284 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 63164.5us
I20260926 01:55:47.718611 9595 raft_consensus.cc:3060] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.741776 9572 raft_consensus.cc:697] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:47.750193 9572 consensus_queue.cc:237] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.757272 9499 consensus_queue.cc:1048] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:47.762840 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04e79a2e52fb40619b2bc99396abda5b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:55:47.765584 9155 leader_election.cc:343] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:47.769800 9523 leader_election.cc:290] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:47.770221 9516 ts_tablet_manager.cc:1929] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:47.770650 9516 log.cc:1199] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/8a54eb3b8cf64a478df38df9d4aef02f
I20260926 01:55:47.774362 9516 ts_tablet_manager.cc:1950] T 8a54eb3b8cf64a478df38df9d4aef02f P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:47.776279 8538 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet b8ec62bc51824766906654d5ab75a095
I20260926 01:55:47.776949 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 8a54eb3b8cf64a478df38df9d4aef02f (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]) successfully deleted
I20260926 01:55:47.777665 9595 raft_consensus.cc:515] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.788234 9514 ts_tablet_manager.cc:1929] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260926 01:55:47.790220 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32e0fe60b8744dbb8ae2bafefe3e834d consensus state has no leader
W20260926 01:55:47.790921 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 119410us
I20260926 01:55:47.793112 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04e79a2e52fb40619b2bc99396abda5b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:47.798215 8907 raft_consensus.cc:3060] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.798869 9515 tablet_replica.cc:333] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:47.799808 9515 raft_consensus.cc:2243] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:47.800374 9515 raft_consensus.cc:2272] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:47.803850 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 562604137d484e83a51de2f778d58fca consensus state has no leader
I20260926 01:55:47.807570 9341 tablet_bootstrap.cc:654] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:47.812027 8907 raft_consensus.cc:2468] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:47.813133 9514 log.cc:1199] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/28bd8695a6b34585bdb1bdabf1bcc654
I20260926 01:55:47.814101 9514 ts_tablet_manager.cc:1950] T 28bd8695a6b34585bdb1bdabf1bcc654 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:47.815088 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 68) took 1593 ms. Trace:
I20260926 01:55:47.815531 9514 rpcz_store.cc:276] 0926 01:55:46.221834 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.222630 (+ 796us) service_pool.cc:224] Handling call
0926 01:55:47.459033 (+1236403us) ts_tablet_manager.cc:1151] Deleting tablet 28bd8695a6b34585bdb1bdabf1bcc654
0926 01:55:47.459080 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:47.521237 (+ 62157us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:47.788075 (+266838us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:47.815023 (+ 26948us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235841}
I20260926 01:55:47.817035 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 28bd8695a6b34585bdb1bdabf1bcc654 (table default.Table_27cdeab15da2426d85cbbb43441e39d6 [id=3b732759201f4eaab4fdf76cd8a38f7e]) successfully deleted
I20260926 01:55:47.795389 9340 tablet_bootstrap.cc:654] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:47.820932 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 182727us
I20260926 01:55:47.822124 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet a85e47c1bf9f4e96958aa2de41039180 with delete_type TABLET_DATA_DELETED (replaced by tablet f1f020590a8b428788e2bb516a11cc45) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:47.822363 9152 leader_election.cc:304] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:47.823156 9523 raft_consensus.cc:2804] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:47.823149 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 099af935249a4310b29852a258661794 (table default.Table_e13d9ec3694c418e906e55984e87a263 [id=0a1b8452d5494c0ea17e1fca2259e171]) successfully deleted
I20260926 01:55:47.823596 9523 raft_consensus.cc:697] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:47.824420 9523 consensus_queue.cc:237] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.818607 9515 ts_tablet_manager.cc:1916] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:47.827759 9532 raft_consensus.cc:2749] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:47.832320 9532 raft_consensus.cc:493] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:47.832782 9532 raft_consensus.cc:515] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.837934 9575 raft_consensus.cc:493] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:47.838547 9575 raft_consensus.cc:515] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.839109 9532 leader_election.cc:290] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:47.840642 9575 leader_election.cc:290] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:47.841439 9341 tablet_bootstrap.cc:492] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:47.841878 9341 ts_tablet_manager.cc:1403] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.234s user 0.009s sys 0.002s
I20260926 01:55:47.845852 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_37930b35bf904b5aabec0d568896d39b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:47.846434 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a54eb3b8cf64a478df38df9d4aef02f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:47.843969 9339 tablet_bootstrap.cc:492] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:47.848366 9339 ts_tablet_manager.cc:1403] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.336s user 0.013s sys 0.006s
I20260926 01:55:47.850780 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_1e95ea0e55dd4033ae92dde6bb3567de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:47.840785 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfff96cc29145e7a72213bb5ecca1ae" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:47.852367 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e95ea0e55dd4033ae92dde6bb3567de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:47.854933 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37930b35bf904b5aabec0d568896d39b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:47.854060 8843 leader_election.cc:343] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 8a54eb3b8cf64a478df38df9d4aef02f
I20260926 01:55:47.857748 9595 leader_election.cc:290] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:47.856256 8843 leader_election.cc:343] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:47.858547 8843 leader_election.cc:304] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:47.859591 9575 raft_consensus.cc:2749] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:47.859330 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_2cabbaf8daaa4f55a14f72b8d40c436f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:47.851502 9072 raft_consensus.cc:3060] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:47.860930 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cabbaf8daaa4f55a14f72b8d40c436f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:47.861205 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dccbfbc9fb5947f9b85ca8560e205b08" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:55:47.870051 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 because the tablet was not found. No further retry: Not found: Tablet not found: 9d9cb4de16da44e1b90ac79e9e32cb88
I20260926 01:55:47.870496 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dccbfbc9fb5947f9b85ca8560e205b08" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:47.871133 8906 raft_consensus.cc:3060] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:47.876358 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 099af935249a4310b29852a258661794 because the tablet was not found. No further retry: Not found: Tablet not found: 099af935249a4310b29852a258661794
W20260926 01:55:47.883401 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 28bd8695a6b34585bdb1bdabf1bcc654 because the tablet was not found. No further retry: Not found: Tablet not found: 28bd8695a6b34585bdb1bdabf1bcc654
I20260926 01:55:47.887429 8906 raft_consensus.cc:2468] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:47.890409 9340 tablet_bootstrap.cc:492] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:47.851563 9339 raft_consensus.cc:359] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.890908 9340 ts_tablet_manager.cc:1403] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.196s user 0.009s sys 0.003s
I20260926 01:55:47.891155 9339 raft_consensus.cc:385] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.891450 9339 raft_consensus.cc:740] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
W20260926 01:55:47.893949 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 3695411d251847b4b1ab0a1eb936a55a because the tablet was not found. No further retry: Not found: Tablet not found: 3695411d251847b4b1ab0a1eb936a55a
I20260926 01:55:47.892174 9339 consensus_queue.cc:260] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.894959 9071 raft_consensus.cc:3060] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:47.896373 9514 tablet_replica.cc:333] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:47.897656 9514 raft_consensus.cc:2243] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:47.899570 9341 raft_consensus.cc:359] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.900286 9341 raft_consensus.cc:385] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.900542 9341 raft_consensus.cc:740] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.901221 9341 consensus_queue.cc:260] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.902473 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dfff96cc29145e7a72213bb5ecca1ae" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:47.903295 9071 raft_consensus.cc:2468] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:47.906795 9155 leader_election.cc:304] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:47.909626 9516 tablet_replica.cc:333] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:47.911536 9006 leader_election.cc:304] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:47.912377 9339 ts_tablet_manager.cc:1434] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.064s user 0.005s sys 0.000s
I20260926 01:55:47.919569 9523 raft_consensus.cc:2804] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:47.920994 9523 raft_consensus.cc:697] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:47.921933 9523 consensus_queue.cc:237] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.933198 9339 tablet_bootstrap.cc:492] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:47.939236 9341 ts_tablet_manager.cc:1434] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.097s user 0.002s sys 0.005s
I20260926 01:55:47.941076 8533 master_service.cc:960] ReplaceTablet: received request to replace tablet 8a011a9edfe94e04afb6ea0367dd979f from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:47.920523 9516 raft_consensus.cc:2243] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:47.942109 9516 raft_consensus.cc:2272] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:47.944038 9514 raft_consensus.cc:2272] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:47.948146 9340 raft_consensus.cc:359] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.948833 9340 raft_consensus.cc:385] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:47.949079 9340 raft_consensus.cc:740] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:47.949720 9340 consensus_queue.cc:260] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.952116 9516 ts_tablet_manager.cc:1916] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:47.953408 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_cf700d4a4dcb4da1afbc424ea51d6f77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:47.952536 9341 tablet_bootstrap.cc:492] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:47.966040 9614 tablet.cc:1751] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:47.968935 9514 ts_tablet_manager.cc:1916] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:47.973047 9339 tablet_bootstrap.cc:654] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:47.983681 8537 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_214d8f2884b048ebba2f07a67d5effe8" } modify_external_catalogs: true
I20260926 01:55:47.984297 8537 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_214d8f2884b048ebba2f07a67d5effe8" } modify_external_catalogs: true
I20260926 01:55:47.989362 9594 raft_consensus.cc:2749] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:47.989838 9499 raft_consensus.cc:493] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:47.990319 9499 raft_consensus.cc:515] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:47.992336 9499 leader_election.cc:290] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:47.994860 8531 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_9c4d9e672f6f409096edc1f1995e23e7" } new_table_name: "default.Table_a8c0c28267ee48f39ee4eb4c11da8a9c" modify_external_catalogs: true
I20260926 01:55:47.997394 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:47.997666 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:47.998268 9219 raft_consensus.cc:2468] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:47.999635 8843 leader_election.cc:304] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
W20260926 01:55:48.000463 8843 leader_election.cc:343] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:48.001513 9575 raft_consensus.cc:2804] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:48.001863 9575 raft_consensus.cc:493] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:48.002170 9575 raft_consensus.cc:3060] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.004920 9341 tablet_bootstrap.cc:654] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.009290 9575 raft_consensus.cc:515] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.013535 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8c9193cf386d4eca9776e2ceb2c7dc11" } new_table_name: "default.Table_a4d2c1ed01c74ff3a7260703edd509be" modify_external_catalogs: true
I20260926 01:55:48.022488 9575 leader_election.cc:290] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.024488 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:48.025066 9219 raft_consensus.cc:3060] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.025904 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:48.026407 9340 ts_tablet_manager.cc:1434] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.135s user 0.004s sys 0.002s
W20260926 01:55:48.027760 8843 leader_election.cc:343] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:48.030571 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf700d4a4dcb4da1afbc424ea51d6f77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.027520 9613 tablet.cc:1751] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:48.043545 9550 raft_consensus.cc:493] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.050366 9550 raft_consensus.cc:515] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.057778 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:48.059665 8843 leader_election.cc:343] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:48.064925 9550 leader_election.cc:290] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.066059 9516 ts_tablet_manager.cc:1929] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.066581 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:48.067181 9218 raft_consensus.cc:2468] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:48.069222 9219 raft_consensus.cc:2468] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:48.073639 9515 ts_tablet_manager.cc:1929] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.074050 9515 log.cc:1199] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/9d10be686d9045efaf384d326b356e0e
I20260926 01:55:48.076521 9341 tablet_bootstrap.cc:492] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:48.076956 9341 ts_tablet_manager.cc:1403] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.127s user 0.008s sys 0.005s
I20260926 01:55:48.079030 9515 ts_tablet_manager.cc:1950] T 9d10be686d9045efaf384d326b356e0e P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:48.068543 8843 leader_election.cc:304] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:48.048775 9340 tablet_bootstrap.cc:492] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:48.080478 8843 leader_election.cc:304] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:48.081544 9617 raft_consensus.cc:2804] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:48.081929 9617 raft_consensus.cc:493] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:48.082332 9617 raft_consensus.cc:3060] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.085644 9618 tablet.cc:1751] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:48.086555 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 9d10be686d9045efaf384d326b356e0e (table default.Table_bf355929f0d5467bb45b71424eb7155b [id=2be262701c0b465f960fc3d666a78026]) successfully deleted
W20260926 01:55:48.088912 8531 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01052a5e22a74a19919bae9dd0296291 consensus state has no leader
I20260926 01:55:48.092070 9515 tablet_replica.cc:333] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:48.093084 9515 raft_consensus.cc:2243] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.093694 9515 raft_consensus.cc:2272] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:48.093868 8536 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04e79a2e52fb40619b2bc99396abda5b consensus state has no leader
I20260926 01:55:48.096189 9341 raft_consensus.cc:359] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.096988 9341 raft_consensus.cc:385] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.097249 9341 raft_consensus.cc:740] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.096027 9550 raft_consensus.cc:2804] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:48.100641 9550 raft_consensus.cc:697] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:48.101570 9550 consensus_queue.cc:237] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.086052 9339 tablet_bootstrap.cc:492] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:48.102520 9516 log.cc:1199] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/a85e47c1bf9f4e96958aa2de41039180
I20260926 01:55:48.099985 9340 tablet_bootstrap.cc:654] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.104225 9339 ts_tablet_manager.cc:1403] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.171s user 0.017s sys 0.006s
I20260926 01:55:48.103950 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_85df772c6e434f49b9b7b7c88949d303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:48.104538 9516 ts_tablet_manager.cc:1950] T a85e47c1bf9f4e96958aa2de41039180 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:55:48.105614 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85df772c6e434f49b9b7b7c88949d303 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.106905 9339 raft_consensus.cc:359] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.107352 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]) successfully deleted
I20260926 01:55:48.107553 9339 raft_consensus.cc:385] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.107800 9339 raft_consensus.cc:740] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.107547 9341 consensus_queue.cc:260] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.109652 9341 ts_tablet_manager.cc:1434] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20260926 01:55:48.117926 9515 ts_tablet_manager.cc:1916] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.119076 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_c74d34ecd1ef4a6d8989b1017deca342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:48.120803 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c74d34ecd1ef4a6d8989b1017deca342 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.126089 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_6f6a2a6f414644f7b5d7a322965922fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:48.128223 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f6a2a6f414644f7b5d7a322965922fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.127799 9514 ts_tablet_manager.cc:1929] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.130793 9514 log.cc:1199] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/099af935249a4310b29852a258661794
I20260926 01:55:48.131667 9514 ts_tablet_manager.cc:1950] T 099af935249a4310b29852a258661794 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:48.133550 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 69) took 1911 ms. Trace:
I20260926 01:55:48.135017 9339 consensus_queue.cc:260] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.137120 9339 ts_tablet_manager.cc:1434] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.033s user 0.004s sys 0.000s
I20260926 01:55:48.148316 9514 rpcz_store.cc:276] 0926 01:55:46.222185 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.225633 (+ 3448us) service_pool.cc:224] Handling call
0926 01:55:47.886381 (+1660748us) ts_tablet_manager.cc:1151] Deleting tablet 099af935249a4310b29852a258661794
0926 01:55:47.886428 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:48.100376 (+213948us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:48.127620 (+ 27244us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:48.133488 (+ 5868us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660160}
I20260926 01:55:48.149783 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 099af935249a4310b29852a258661794 (table default.Table_e13d9ec3694c418e906e55984e87a263 [id=0a1b8452d5494c0ea17e1fca2259e171]) successfully deleted
I20260926 01:55:48.150125 9514 tablet_replica.cc:333] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:48.151026 9514 raft_consensus.cc:2243] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:48.156250 9339 tablet_bootstrap.cc:492] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:48.162542 9514 raft_consensus.cc:2272] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.169680 9515 ts_tablet_manager.cc:1929] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.166261 9514 ts_tablet_manager.cc:1916] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:48.171622 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet f3bb1643da054583b97efd2e1a9e9c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bb1643da054583b97efd2e1a9e9c2f already in progress: creating tablet
I20260926 01:55:48.171787 9515 log.cc:1199] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/28bd8695a6b34585bdb1bdabf1bcc654
I20260926 01:55:48.172394 9339 tablet_bootstrap.cc:654] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.173487 9341 tablet_bootstrap.cc:492] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:48.173913 9515 ts_tablet_manager.cc:1950] T 28bd8695a6b34585bdb1bdabf1bcc654 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:48.186025 9341 tablet_bootstrap.cc:654] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.188304 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 28bd8695a6b34585bdb1bdabf1bcc654 (table default.Table_27cdeab15da2426d85cbbb43441e39d6 [id=3b732759201f4eaab4fdf76cd8a38f7e]) successfully deleted
W20260926 01:55:48.190128 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet f3bb1643da054583b97efd2e1a9e9c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bb1643da054583b97efd2e1a9e9c2f already in progress: creating tablet
I20260926 01:55:48.191314 9515 tablet_replica.cc:333] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:48.192118 9515 raft_consensus.cc:2243] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.192679 9515 raft_consensus.cc:2272] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.193055 9339 tablet_bootstrap.cc:492] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:48.194810 9339 ts_tablet_manager.cc:1403] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.001s
I20260926 01:55:48.162984 9340 tablet_bootstrap.cc:492] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:48.196364 9340 ts_tablet_manager.cc:1403] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.148s user 0.010s sys 0.003s
I20260926 01:55:48.197525 9515 ts_tablet_manager.cc:1916] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.197355 9339 raft_consensus.cc:359] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.201069 9339 raft_consensus.cc:385] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.201356 9339 raft_consensus.cc:740] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.202108 9339 consensus_queue.cc:260] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.204238 9339 ts_tablet_manager.cc:1434] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260926 01:55:48.198926 9340 raft_consensus.cc:359] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.210770 9340 raft_consensus.cc:385] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.211063 9340 raft_consensus.cc:740] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.157277 9617 raft_consensus.cc:515] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.212141 9339 tablet_bootstrap.cc:492] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:48.213052 9617 leader_election.cc:290] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.213707 9340 consensus_queue.cc:260] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.215790 9340 ts_tablet_manager.cc:1434] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260926 01:55:48.219281 9339 tablet_bootstrap.cc:654] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.223443 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:48.224082 9219 raft_consensus.cc:3060] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.228777 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:48.235837 8535 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_d7233ccd12524b5eae620cd58c593335 [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 2 got 0
I20260926 01:55:48.235841 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f (DEFAULT_TABLE table=default.Table_235a08a4f73a4761935c03af88c3a393 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.236905 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cba678b66744dcfb4d13ae07ebec33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.238284 9340 tablet_bootstrap.cc:492] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:48.237149 8540 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63] alter complete (version 1)
I20260926 01:55:48.238227 9194 tablet_service.cc:1511] Processing CreateTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f (DEFAULT_TABLE table=default.Table_235a08a4f73a4761935c03af88c3a393 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.239460 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cba678b66744dcfb4d13ae07ebec33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.239545 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f (DEFAULT_TABLE table=default.Table_235a08a4f73a4761935c03af88c3a393 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.240551 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cba678b66744dcfb4d13ae07ebec33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.248324 9195 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f with delete_type TABLET_DATA_DELETED (replaced by tablet 3571967749844fd7859cf179a93cf403) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:48.249101 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:48.249318 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:48.250808 9516 tablet_replica.cc:333] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:48.251530 9516 raft_consensus.cc:2243] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:48.252209 9516 pending_rounds.cc:70] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Trying to abort 1 pending ops.
I20260926 01:55:48.252401 9516 pending_rounds.cc:77] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428215497641984 op_type: NO_OP noop_request { }
I20260926 01:55:48.252701 9516 raft_consensus.cc:2889] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:48.253002 9516 raft_consensus.cc:2272] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:48.253139 8525 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 01052a5e22a74a19919bae9dd0296291 (table default.Table_a8c0c28267ee48f39ee4eb4c11da8a9c [id=6a3cae01c8664f3d8d4b89e8a91665f7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 01052a5e22a74a19919bae9dd0296291 consensus state has no leader
I20260926 01:55:48.249102 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet a85e47c1bf9f4e96958aa2de41039180 with delete_type TABLET_DATA_DELETED (replaced by tablet f1f020590a8b428788e2bb516a11cc45) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:48.255147 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 8a54eb3b8cf64a478df38df9d4aef02f because the tablet was not found. No further retry: Not found: Tablet not found: 8a54eb3b8cf64a478df38df9d4aef02f
W20260926 01:55:48.255848 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 9d10be686d9045efaf384d326b356e0e because the tablet was not found. No further retry: Not found: Tablet not found: 9d10be686d9045efaf384d326b356e0e
W20260926 01:55:48.258216 8539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 34) took 1143 ms (client timeout 1000 ms). Trace:
I20260926 01:55:48.259832 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:48.260887 9516 ts_tablet_manager.cc:1916] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.262233 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:48.258654 8539 rpcz_store.cc:269] 0926 01:55:47.114732 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:47.114913 (+ 181us) service_pool.cc:224] Handling call
0926 01:55:48.258145 (+1143232us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:47.608788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:47.609191 (+ 403us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:47.609217 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:47.609229 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:47.609494 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:47.609534 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:47.609558 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:47.609728 (+ 170us) tablet.cc:701] Row locks acquired
0926 01:55:47.609742 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:47.609862 (+ 120us) write_op.cc:270] Start()
0926 01:55:47.609950 (+ 88us) write_op.cc:276] Timestamp: P: 1790387747609816 usec, L: 0
0926 01:55:47.609964 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:47.610696 (+ 732us) log.cc:844] Serialized 530 byte log entry
0926 01:55:47.724588 (+113892us) op_driver.cc:464] REPLICATION: finished
0926 01:55:47.724833 (+ 245us) write_op.cc:301] APPLY: starting
0926 01:55:47.724890 (+ 57us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:47.724993 (+ 103us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:47.725076 (+ 83us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:47.725247 (+ 171us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:47.725373 (+ 126us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:47.725413 (+ 40us) write_op.cc:312] APPLY: finished
0926 01:55:47.725616 (+ 203us) log.cc:844] Serialized 23 byte log entry
0926 01:55:47.725765 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:47.725830 (+ 65us) write_op.cc:454] Released schema lock
0926 01:55:47.725861 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":1215,"apply.run_wall_time_us":1216,"num_ops":1,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":2028,"prepare.run_wall_time_us":2205,"raft.queue_time_us":172,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":114526}]]}
I20260926 01:55:48.265959 9341 tablet_bootstrap.cc:492] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:48.266389 9341 ts_tablet_manager.cc:1403] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.000s
I20260926 01:55:48.268599 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:48.269892 8843 leader_election.cc:343] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:48.277951 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet 099af935249a4310b29852a258661794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:48.280287 9339 tablet_bootstrap.cc:492] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:48.281704 9339 ts_tablet_manager.cc:1403] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.005s
I20260926 01:55:48.282991 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet 28bd8695a6b34585bdb1bdabf1bcc654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:48.284264 9339 raft_consensus.cc:359] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.284874 9339 raft_consensus.cc:385] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.285126 9339 raft_consensus.cc:740] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.285310 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:48.285986 9339 consensus_queue.cc:260] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.287971 9339 ts_tablet_manager.cc:1434] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 01:55:48.288908 9339 tablet_bootstrap.cc:492] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:48.297336 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d7233ccd12524b5eae620cd58c593335" } new_table_name: "default.Table_e313053803ad4a608bc4a5a546458c5d" modify_external_catalogs: true
I20260926 01:55:48.298631 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:48.301328 9340 tablet_bootstrap.cc:654] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:48.305537 8535 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 34) took 839 ms (client timeout 1000 ms). Trace:
I20260926 01:55:48.305245 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_3478e66c0ff74bc4a27eb418e346e4e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:48.305977 8535 rpcz_store.cc:269] 0926 01:55:47.465894 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:47.468270 (+ 2376us) service_pool.cc:224] Handling call
0926 01:55:47.478442 (+ 10172us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:48.305473 (+827031us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1843,"reported_tablets":9,"spinlock_wait_cycles":18807680}
W20260926 01:55:48.307011 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3478e66c0ff74bc4a27eb418e346e4e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.314234 9339 tablet_bootstrap.cc:654] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.318529 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_2cabbaf8daaa4f55a14f72b8d40c436f" } new_table_name: "default.Table_3907f5fafff14aec94d0a9e331e86dc9" modify_external_catalogs: true
I20260926 01:55:48.333222 9339 tablet_bootstrap.cc:492] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:48.334301 9339 ts_tablet_manager.cc:1403] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.003s
I20260926 01:55:48.337014 9515 ts_tablet_manager.cc:1929] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.340869 9515 log.cc:1199] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/a85e47c1bf9f4e96958aa2de41039180
I20260926 01:55:48.341758 9515 ts_tablet_manager.cc:1950] T a85e47c1bf9f4e96958aa2de41039180 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:48.341706 9514 ts_tablet_manager.cc:1929] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.337061 9339 raft_consensus.cc:359] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.342146 9514 log.cc:1199] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/9d10be686d9045efaf384d326b356e0e
I20260926 01:55:48.342555 9339 raft_consensus.cc:385] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.342927 9514 ts_tablet_manager.cc:1950] T 9d10be686d9045efaf384d326b356e0e P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:48.343204 9339 raft_consensus.cc:740] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.345310 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]) successfully deleted
W20260926 01:55:48.348018 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 9d10be686d9045efaf384d326b356e0e because the tablet was not found. No further retry: Not found: Tablet not found: 9d10be686d9045efaf384d326b356e0e
I20260926 01:55:48.344661 9339 consensus_queue.cc:260] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:48.349961 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 28bd8695a6b34585bdb1bdabf1bcc654 because the tablet was not found. No further retry: Not found: Tablet not found: 28bd8695a6b34585bdb1bdabf1bcc654
W20260926 01:55:48.352178 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 099af935249a4310b29852a258661794 because the tablet was not found. No further retry: Not found: Tablet not found: 099af935249a4310b29852a258661794
I20260926 01:55:48.353066 9515 tablet_replica.cc:333] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:48.361263 9339 ts_tablet_manager.cc:1434] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260926 01:55:48.362025 9515 raft_consensus.cc:2243] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.362682 9515 raft_consensus.cc:2272] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:48.362735 8843 consensus_peers.cc:597] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85e47c1bf9f4e96958aa2de41039180. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:48.363776 9339 tablet_bootstrap.cc:492] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:48.366420 9515 ts_tablet_manager.cc:1916] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.368463 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 70) took 2113 ms. Trace:
I20260926 01:55:48.368070 9340 tablet_bootstrap.cc:492] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:48.374020 9340 ts_tablet_manager.cc:1403] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.136s user 0.015s sys 0.000s
I20260926 01:55:48.375048 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b205886d445546d08910c2f6fb5db3c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:48.368872 9595 raft_consensus.cc:493] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.379563 9340 raft_consensus.cc:359] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.380214 9340 raft_consensus.cc:385] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.380481 9340 raft_consensus.cc:740] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.381152 9340 consensus_queue.cc:260] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.389127 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 (DEFAULT_TABLE table=default.Table_cdcfa57f52a747aaa93bb04410df7061 [id=fccbbbc294bb4548813a1229bd9e2ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.376858 8534 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:48.390306 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55452e70a7fe48efbafdc7cbb29c6272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.390884 9341 raft_consensus.cc:359] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.391523 9341 raft_consensus.cc:385] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.391549 8534 catalog_manager.cc:6934] ReplaceTablet: tablet 32e0fe60b8744dbb8ae2bafefe3e834d deleted and replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b
I20260926 01:55:48.391767 9341 raft_consensus.cc:740] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.392575 9516 ts_tablet_manager.cc:1929] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.392933 9516 log.cc:1199] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:48.369196 9514 rpcz_store.cc:276] 0926 01:55:46.255540 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.255668 (+ 128us) service_pool.cc:224] Handling call
0926 01:55:48.149931 (+1894263us) ts_tablet_manager.cc:1151] Deleting tablet 9d10be686d9045efaf384d326b356e0e
0926 01:55:48.149979 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:48.199546 (+ 49567us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:48.341496 (+141950us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:48.368392 (+ 26896us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15744,"tablet-delete.queue_time_us":1893734}
I20260926 01:55:48.393590 9516 ts_tablet_manager.cc:1950] T dccbfbc9fb5947f9b85ca8560e205b08 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:48.394277 9219 raft_consensus.cc:2468] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:48.394394 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 77) took 1440 ms. Trace:
I20260926 01:55:48.394778 9514 rpcz_store.cc:276] 0926 01:55:46.954056 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.954179 (+ 123us) service_pool.cc:224] Handling call
0926 01:55:48.393814 (+1439635us) ts_tablet_manager.cc:1151] Deleting tablet 75d4598596ff49378fc3536fff0ac1b1
0926 01:55:48.393863 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:48.394324 (+ 461us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1383496}
W20260926 01:55:48.395682 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b205886d445546d08910c2f6fb5db3c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.396369 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 78) took 1429 ms. Trace:
I20260926 01:55:48.380124 9595 raft_consensus.cc:515] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.396673 9514 rpcz_store.cc:276] 0926 01:55:46.966938 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:46.967059 (+ 121us) service_pool.cc:224] Handling call
0926 01:55:48.396003 (+1428944us) ts_tablet_manager.cc:1151] Deleting tablet 1866cd97bb024a9baecf101a6d5b1ee4
0926 01:55:48.396051 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:48.396314 (+ 263us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1385506}
I20260926 01:55:48.397980 9514 tablet_replica.cc:333] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:48.398770 9514 raft_consensus.cc:2243] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:48.398823 9595 leader_election.cc:290] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:48.399600 9514 raft_consensus.cc:2272] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.400897 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1866cd97bb024a9baecf101a6d5b1ee4" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:48.401494 8907 raft_consensus.cc:2468] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:48.403198 9340 ts_tablet_manager.cc:1434] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260926 01:55:48.403260 9152 leader_election.cc:304] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:48.404628 9340 tablet_bootstrap.cc:492] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:48.410969 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1866cd97bb024a9baecf101a6d5b1ee4" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:48.412400 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:48.412470 8843 consensus_peers.cc:597] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a85e47c1bf9f4e96958aa2de41039180. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:48.413141 8843 leader_election.cc:304] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:48.413686 9594 raft_consensus.cc:493] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.414398 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:48.414158 9594 raft_consensus.cc:515] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.414732 9499 raft_consensus.cc:2804] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:48.415180 9499 raft_consensus.cc:697] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:48.416373 9594 leader_election.cc:290] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.416003 9499 consensus_queue.cc:237] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:48.417109 9155 leader_election.cc:343] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:48.418138 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet dccbfbc9fb5947f9b85ca8560e205b08 (table default.Table_14869892bcf7429385632e2df181743c [id=bfa8586ef17949bead9731402cd78042]) successfully deleted
W20260926 01:55:48.418896 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet a85e47c1bf9f4e96958aa2de41039180 because the tablet was not found. No further retry: Not found: Tablet not found: a85e47c1bf9f4e96958aa2de41039180
I20260926 01:55:48.419353 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04e79a2e52fb40619b2bc99396abda5b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:48.419486 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_c30c84e91ae143e3b6918b9be1ec643e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:48.420087 8907 raft_consensus.cc:2393] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:48.421222 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04e79a2e52fb40619b2bc99396abda5b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:55:48.421409 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c30c84e91ae143e3b6918b9be1ec643e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.422887 9006 leader_election.cc:304] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:48.423305 9339 tablet_bootstrap.cc:654] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:48.423971 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 32e0fe60b8744dbb8ae2bafefe3e834d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32e0fe60b8744dbb8ae2bafefe3e834d already in progress: creating tablet
I20260926 01:55:48.424799 9594 raft_consensus.cc:3060] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.428431 9595 raft_consensus.cc:2804] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:48.428830 9595 raft_consensus.cc:493] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:48.429128 9595 raft_consensus.cc:3060] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.430684 9341 consensus_queue.cc:260] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.434000 9594 raft_consensus.cc:2749] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:48.445155 9341 ts_tablet_manager.cc:1434] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.178s user 0.007s sys 0.000s
I20260926 01:55:48.448101 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:48.449087 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 (DEFAULT_TABLE table=default.Table_cdcfa57f52a747aaa93bb04410df7061 [id=fccbbbc294bb4548813a1229bd9e2ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.450287 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55452e70a7fe48efbafdc7cbb29c6272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.459313 9341 tablet_bootstrap.cc:492] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:48.465807 9339 tablet_bootstrap.cc:492] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:48.466605 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 2 got 0
I20260926 01:55:48.467337 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:48.475399 9340 tablet_bootstrap.cc:654] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.478708 9339 ts_tablet_manager.cc:1403] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.115s user 0.016s sys 0.000s
I20260926 01:55:48.479135 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet dfaca3fc02634f459083d23f3102f725
I20260926 01:55:48.486503 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet dfaca3fc02634f459083d23f3102f725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:48.487438 9516 tablet_replica.cc:333] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:48.488338 9516 raft_consensus.cc:2243] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.488866 9516 raft_consensus.cc:2272] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.489226 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet dfaca3fc02634f459083d23f3102f725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:48.491956 9516 ts_tablet_manager.cc:1916] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.494228 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 (DEFAULT_TABLE table=default.Table_92899e57b6c14f62b2a999bb2487f558 [id=45e5e00e818749a8beb091f6a62b7bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.401455 9514 ts_tablet_manager.cc:1916] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.495359 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cd9130640d4b14beef42e9f1b36b99. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:48.502017 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 54848.8us
I20260926 01:55:48.503590 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 9d10be686d9045efaf384d326b356e0e (table default.Table_bf355929f0d5467bb45b71424eb7155b [id=2be262701c0b465f960fc3d666a78026]) successfully deleted
W20260926 01:55:48.504596 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 75d4598596ff49378fc3536fff0ac1b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75d4598596ff49378fc3536fff0ac1b1 already in progress: creating tablet
I20260926 01:55:48.510669 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 (DEFAULT_TABLE table=default.Table_cdcfa57f52a747aaa93bb04410df7061 [id=fccbbbc294bb4548813a1229bd9e2ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.511354 9341 tablet_bootstrap.cc:654] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.492336 9339 raft_consensus.cc:359] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.511885 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55452e70a7fe48efbafdc7cbb29c6272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.512152 9339 raft_consensus.cc:385] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.512444 9339 raft_consensus.cc:740] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.516444 9340 tablet_bootstrap.cc:492] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:48.516906 9340 ts_tablet_manager.cc:1403] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.112s user 0.010s sys 0.000s
I20260926 01:55:48.522751 9339 consensus_queue.cc:260] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.528476 9339 ts_tablet_manager.cc:1434] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260926 01:55:48.529394 9339 tablet_bootstrap.cc:492] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:48.535746 9516 ts_tablet_manager.cc:1929] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:48.539417 9340 raft_consensus.cc:359] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.540114 9340 raft_consensus.cc:385] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.540385 9340 raft_consensus.cc:740] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.509171 9595 raft_consensus.cc:515] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.546118 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1866cd97bb024a9baecf101a6d5b1ee4" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:48.546744 8907 raft_consensus.cc:3060] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.549371 9339 tablet_bootstrap.cc:654] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.544687 9340 consensus_queue.cc:260] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.554397 8907 raft_consensus.cc:2468] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:48.555444 9595 leader_election.cc:290] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:48.556860 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1866cd97bb024a9baecf101a6d5b1ee4" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:48.557096 9152 leader_election.cc:304] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:48.557891 9595 raft_consensus.cc:2804] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:48.558408 9523 raft_consensus.cc:493] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.558660 9516 log.cc:1199] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/dfaca3fc02634f459083d23f3102f725
I20260926 01:55:48.558892 9523 raft_consensus.cc:515] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.559619 9516 ts_tablet_manager.cc:1950] T dfaca3fc02634f459083d23f3102f725 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:48.560927 9523 leader_election.cc:290] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:48.562172 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b6daaf274894989b90da364f5da5250" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:48.562393 9053 tablet_service.cc:1511] Processing CreateTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b (DEFAULT_TABLE table=default.Table_29ce250e9d9a4162966731d01a8e3b23 [id=e664d2aa0cf74dfbad934ad658e8ec0e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:48.563702 9152 leader_election.cc:343] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:48.563998 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27f1dd7bb0449a6a3fa371a8636e09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.565778 9594 raft_consensus.cc:493] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.566247 9594 raft_consensus.cc:515] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.568668 9594 leader_election.cc:290] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.569098 9193 tablet_service.cc:1511] Processing CreateTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 (DEFAULT_TABLE table=default.Table_92899e57b6c14f62b2a999bb2487f558 [id=45e5e00e818749a8beb091f6a62b7bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.569312 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:48.570308 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cd9130640d4b14beef42e9f1b36b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.570850 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b6daaf274894989b90da364f5da5250" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:48.575099 9523 consensus_queue.cc:1048] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:48.575286 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03086ec5451743fdbf2d5726d43f31b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:48.576177 9220 raft_consensus.cc:2393] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
W20260926 01:55:48.569278 9155 leader_election.cc:343] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:48.577527 9155 leader_election.cc:343] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:48.577515 9006 leader_election.cc:304] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:48.577911 9155 leader_election.cc:304] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:48.579161 9523 raft_consensus.cc:2749] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:48.585860 9594 raft_consensus.cc:3060] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.580404 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet dfaca3fc02634f459083d23f3102f725 (table default.Table_214d8f2884b048ebba2f07a67d5effe8 [id=bcf837a98f554432b1f529f1c6c5c4c0]) successfully deleted
I20260926 01:55:48.587024 9595 raft_consensus.cc:697] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
W20260926 01:55:48.586834 8524 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 82261.9us
W20260926 01:55:48.588138 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 1866cd97bb024a9baecf101a6d5b1ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1866cd97bb024a9baecf101a6d5b1ee4 already in progress: creating tablet
I20260926 01:55:48.567716 9340 ts_tablet_manager.cc:1434] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20260926 01:55:48.592993 9340 tablet_bootstrap.cc:492] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:48.588865 9595 consensus_queue.cc:237] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.584991 8537 catalog_manager.cc:5838] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:48.596825 8537 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 2 got 0
I20260926 01:55:48.599100 8537 catalog_manager.cc:5838] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:48.604720 8537 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_a4d2c1ed01c74ff3a7260703edd509be [id=2af856781c904c77a11ca9f4966ca44b]). Expected version 3 got 0
I20260926 01:55:48.605489 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet dfaca3fc02634f459083d23f3102f725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:48.608777 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 (DEFAULT_TABLE table=default.Table_92899e57b6c14f62b2a999bb2487f558 [id=45e5e00e818749a8beb091f6a62b7bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.609968 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39cd9130640d4b14beef42e9f1b36b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.615216 8537 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:48.616179 9341 tablet_bootstrap.cc:492] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:48.616223 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 3b6daaf274894989b90da364f5da5250
I20260926 01:55:48.616617 9341 ts_tablet_manager.cc:1403] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.158s user 0.007s sys 0.005s
I20260926 01:55:48.617403 8533 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 8a011a9edfe94e04afb6ea0367dd979f
I20260926 01:55:48.618748 8533 catalog_manager.cc:6934] ReplaceTablet: tablet 8a011a9edfe94e04afb6ea0367dd979f deleted and replaced by tablet 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:55:48.620877 9594 raft_consensus.cc:2749] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:48.620959 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b6daaf274894989b90da364f5da5250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:48.621680 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f with delete_type TABLET_DATA_DELETED (replaced by tablet 17b8b0d54f5645e5963f1de217f94c84) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:48.621630 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f with delete_type TABLET_DATA_DELETED (replaced by tablet 17b8b0d54f5645e5963f1de217f94c84) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:48.622339 9516 tablet_replica.cc:333] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:48.623071 9516 raft_consensus.cc:2243] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.623183 9515 ts_tablet_manager.cc:1929] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.623574 9516 raft_consensus.cc:2272] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.624941 9341 raft_consensus.cc:359] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.625581 9341 raft_consensus.cc:385] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.625698 9516 ts_tablet_manager.cc:1916] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.625849 9341 raft_consensus.cc:740] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:55:48.611768 8524 consensus_peers.cc:597] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba -> Peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Couldn't send request to peer f75ba46482de422ea23daf17aa100849. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260926 01:55:48.629060 9070 raft_consensus.cc:3060] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.629711 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b6daaf274894989b90da364f5da5250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:48.636024 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_87c331af29e7419593789c486dc50a13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:48.638736 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87c331af29e7419593789c486dc50a13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:48.641521 9595 consensus_queue.cc:1048] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:48.623514 9515 log.cc:1199] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/3695411d251847b4b1ab0a1eb936a55a
I20260926 01:55:48.647740 9515 ts_tablet_manager.cc:1950] T 3695411d251847b4b1ab0a1eb936a55a P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:48.649643 9341 consensus_queue.cc:260] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.652070 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 3695411d251847b4b1ab0a1eb936a55a (table default.Table_37ba6de1661d4b1f95e93ea4b3348633 [id=a2b0b9ebe53d44bd99b0eae816b717fb]) successfully deleted
I20260926 01:55:48.660149 9341 ts_tablet_manager.cc:1434] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.043s user 0.005s sys 0.002s
W20260926 01:55:48.662734 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 32e0fe60b8744dbb8ae2bafefe3e834d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32e0fe60b8744dbb8ae2bafefe3e834d already in progress: creating tablet
W20260926 01:55:48.665416 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet dfaca3fc02634f459083d23f3102f725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfaca3fc02634f459083d23f3102f725 already in progress: creating tablet
W20260926 01:55:48.668006 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 3b6daaf274894989b90da364f5da5250 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b6daaf274894989b90da364f5da5250 already in progress: creating tablet
I20260926 01:55:48.672608 9515 tablet_replica.cc:333] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:48.672914 9341 tablet_bootstrap.cc:492] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:48.675786 9515 raft_consensus.cc:2243] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.675868 9516 ts_tablet_manager.cc:1929] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:48.676244 9516 log.cc:1199] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/8a011a9edfe94e04afb6ea0367dd979f
I20260926 01:55:48.676343 9193 tablet_service.cc:1511] Processing CreateTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b (DEFAULT_TABLE table=default.Table_29ce250e9d9a4162966731d01a8e3b23 [id=e664d2aa0cf74dfbad934ad658e8ec0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.677028 9516 ts_tablet_manager.cc:1950] T 8a011a9edfe94e04afb6ea0367dd979f P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:48.677371 9515 raft_consensus.cc:2272] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.677628 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27f1dd7bb0449a6a3fa371a8636e09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.679848 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]) successfully deleted
I20260926 01:55:48.682201 9341 tablet_bootstrap.cc:654] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.683562 9198 tablet_service.cc:1511] Processing CreateTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 (DEFAULT_TABLE table=default.Table_722ce830650d49ffa0b00028e056e1db [id=b10f8555970c4add9293d27073a1175b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.689641 8533 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_17296161bc2343d1acf54e54e6e59e3b" } modify_external_catalogs: true
I20260926 01:55:48.690232 8533 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_17296161bc2343d1acf54e54e6e59e3b" } modify_external_catalogs: true
I20260926 01:55:48.690448 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36ea09cf15647c0a344c9ba9b289d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.694757 9516 tablet_replica.cc:333] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:48.703078 8878 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b6daaf274894989b90da364f5da5250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:48.704294 8884 tablet_service.cc:1511] Processing CreateTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b (DEFAULT_TABLE table=default.Table_29ce250e9d9a4162966731d01a8e3b23 [id=e664d2aa0cf74dfbad934ad658e8ec0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.705637 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27f1dd7bb0449a6a3fa371a8636e09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.711759 9617 raft_consensus.cc:493] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.712252 9617 raft_consensus.cc:515] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.712957 8878 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f with delete_type TABLET_DATA_DELETED (replaced by tablet 17b8b0d54f5645e5963f1de217f94c84) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:48.714118 9617 leader_election.cc:290] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.715332 9340 tablet_bootstrap.cc:654] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.718557 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659eca3c97144aaacd41e747aecd96d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:48.719069 9516 raft_consensus.cc:2243] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.719239 9218 raft_consensus.cc:2468] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:48.719712 9516 raft_consensus.cc:2272] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.720881 8843 leader_election.cc:304] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:48.721633 9617 raft_consensus.cc:2804] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:48.721956 9617 raft_consensus.cc:493] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:48.721854 8878 tablet_service.cc:1511] Processing CreateTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 (DEFAULT_TABLE table=default.Table_722ce830650d49ffa0b00028e056e1db [id=b10f8555970c4add9293d27073a1175b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.722295 9617 raft_consensus.cc:3060] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.723745 9516 ts_tablet_manager.cc:1916] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.725215 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36ea09cf15647c0a344c9ba9b289d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.726379 9514 ts_tablet_manager.cc:1929] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:48.731019 9514 log.cc:1199] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/3695411d251847b4b1ab0a1eb936a55a
I20260926 01:55:48.717231 9595 consensus_queue.cc:1048] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:48.736716 9514 ts_tablet_manager.cc:1950] T 3695411d251847b4b1ab0a1eb936a55a P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:48.743325 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 82) took 1709 ms. Trace:
I20260926 01:55:48.746569 9514 rpcz_store.cc:276] 0926 01:55:47.034336 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:47.034880 (+ 544us) service_pool.cc:224] Handling call
0926 01:55:48.397819 (+1362939us) ts_tablet_manager.cc:1151] Deleting tablet 3695411d251847b4b1ab0a1eb936a55a
0926 01:55:48.397861 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:48.526159 (+128298us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:48.718313 (+192154us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:48.743260 (+ 24947us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362240}
I20260926 01:55:48.748360 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 3695411d251847b4b1ab0a1eb936a55a (table default.Table_37ba6de1661d4b1f95e93ea4b3348633 [id=a2b0b9ebe53d44bd99b0eae816b717fb]) successfully deleted
I20260926 01:55:48.751349 9617 raft_consensus.cc:515] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.678403 9050 tablet_service.cc:1511] Processing CreateTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 (DEFAULT_TABLE table=default.Table_722ce830650d49ffa0b00028e056e1db [id=b10f8555970c4add9293d27073a1175b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.760440 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36ea09cf15647c0a344c9ba9b289d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.755038 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 83) took 1635 ms. Trace:
I20260926 01:55:48.761430 9514 rpcz_store.cc:276] 0926 01:55:47.134631 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:47.134763 (+ 132us) service_pool.cc:224] Handling call
0926 01:55:48.754709 (+1619946us) ts_tablet_manager.cc:1151] Deleting tablet 03b827807def43b39b91d61a6dfd9386
0926 01:55:48.754773 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:48.754974 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1619225}
I20260926 01:55:48.794413 9635 consensus_queue.cc:1048] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:48.806865 9052 tablet_service.cc:1511] Processing CreateTablet for tablet ee418347ebb04927913810d539ef57ea (DEFAULT_TABLE table=default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.807610 8884 tablet_service.cc:1511] Processing CreateTablet for tablet ee418347ebb04927913810d539ef57ea (DEFAULT_TABLE table=default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:48.815699 9636 log.cc:927] Time spent T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260926 01:55:48.816515 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 03b827807def43b39b91d61a6dfd9386 because the tablet was not found. No further retry: Not found: Tablet not found: 03b827807def43b39b91d61a6dfd9386
I20260926 01:55:48.817816 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee418347ebb04927913810d539ef57ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.824898 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659eca3c97144aaacd41e747aecd96d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:48.825505 9218 raft_consensus.cc:3060] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.825619 9340 tablet_bootstrap.cc:492] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:48.826016 9340 ts_tablet_manager.cc:1403] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.233s user 0.008s sys 0.008s
I20260926 01:55:48.834090 9218 raft_consensus.cc:2468] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:48.835863 9617 leader_election.cc:290] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.845324 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee418347ebb04927913810d539ef57ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.845592 9515 ts_tablet_manager.cc:1916] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.847326 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659eca3c97144aaacd41e747aecd96d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:48.847616 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b659eca3c97144aaacd41e747aecd96d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:48.848011 9073 raft_consensus.cc:2468] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:48.855173 9193 tablet_service.cc:1511] Processing CreateTablet for tablet ee418347ebb04927913810d539ef57ea (DEFAULT_TABLE table=default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.857239 9047 tablet_service.cc:1511] Processing CreateTablet for tablet f2ee16b0c3e84936abba8248f194786a (DEFAULT_TABLE table=default.Table_2b020c7bf3ca40f5932fadfe0dc566d4 [id=182d678afe7743aba274cc33ba1a445c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.859355 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee16b0c3e84936abba8248f194786a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.857151 9341 tablet_bootstrap.cc:492] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:48.862172 9341 ts_tablet_manager.cc:1403] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.193s user 0.012s sys 0.002s
I20260926 01:55:48.864084 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee418347ebb04927913810d539ef57ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.868623 9194 tablet_service.cc:1511] Processing CreateTablet for tablet f2ee16b0c3e84936abba8248f194786a (DEFAULT_TABLE table=default.Table_2b020c7bf3ca40f5932fadfe0dc566d4 [id=182d678afe7743aba274cc33ba1a445c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.870007 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee16b0c3e84936abba8248f194786a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.867090 9341 raft_consensus.cc:359] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.874837 9341 raft_consensus.cc:385] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.875124 9341 raft_consensus.cc:740] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.879391 9341 consensus_queue.cc:260] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.885309 9341 ts_tablet_manager.cc:1434] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260926 01:55:48.898164 9341 tablet_bootstrap.cc:492] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:48.912091 9341 tablet_bootstrap.cc:654] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:48.920095 8540 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_722ce830650d49ffa0b00028e056e1db" } modify_external_catalogs: true
I20260926 01:55:48.920682 8540 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_722ce830650d49ffa0b00028e056e1db" } modify_external_catalogs: true
W20260926 01:55:48.927578 8536 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 943a990fa10948e8961c02e9999569f5 has no consensus state
I20260926 01:55:48.928714 8536 master_service.cc:960] ReplaceTablet: received request to replace tablet 1cba678b66744dcfb4d13ae07ebec33f from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:48.816256 9514 tablet_replica.cc:333] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:48.843957 8843 leader_election.cc:304] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
W20260926 01:55:48.940292 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 96469.4us
I20260926 01:55:48.867841 9499 raft_consensus.cc:493] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.942682 9499 raft_consensus.cc:515] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.943887 9575 raft_consensus.cc:2804] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:48.944613 9499 leader_election.cc:290] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:48.949074 8843 consensus_peers.cc:597] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:48.952513 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:48.957821 9220 raft_consensus.cc:2468] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:48.958252 8534 master_service.cc:960] ReplaceTablet: received request to replace tablet a85e47c1bf9f4e96958aa2de41039180 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:48.960021 9514 raft_consensus.cc:2243] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:48.960520 9514 raft_consensus.cc:2272] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:48.961349 8843 leader_election.cc:304] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:48.962100 9499 raft_consensus.cc:2804] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:48.879920 9340 raft_consensus.cc:359] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.962786 9340 raft_consensus.cc:385] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.962781 9514 ts_tablet_manager.cc:1916] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:48.963128 9340 raft_consensus.cc:740] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.962442 9499 raft_consensus.cc:493] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:48.964318 9499 raft_consensus.cc:3060] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:48.965833 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:48.963999 9340 consensus_queue.cc:260] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.968938 9340 ts_tablet_manager.cc:1434] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.143s user 0.006s sys 0.001s
I20260926 01:55:48.970358 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 8a56044249bb4aaa81903d2a8aeb008e (DEFAULT_TABLE table=default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.969803 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 8a56044249bb4aaa81903d2a8aeb008e (DEFAULT_TABLE table=default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.971511 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a56044249bb4aaa81903d2a8aeb008e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.971805 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a56044249bb4aaa81903d2a8aeb008e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.974440 9341 tablet_bootstrap.cc:492] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:48.975032 9594 raft_consensus.cc:493] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.976390 9594 raft_consensus.cc:515] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:48.979188 9594 leader_election.cc:290] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:48.980738 9341 ts_tablet_manager.cc:1403] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.000s
I20260926 01:55:48.979804 9516 ts_tablet_manager.cc:1929] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:48.982159 9516 log.cc:1199] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/3b6daaf274894989b90da364f5da5250
I20260926 01:55:48.983323 9516 ts_tablet_manager.cc:1950] T 3b6daaf274894989b90da364f5da5250 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:55:48.983619 9006 leader_election.cc:343] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 8a54eb3b8cf64a478df38df9d4aef02f
I20260926 01:55:48.989912 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 3b6daaf274894989b90da364f5da5250 (table default.Table_efe5a3dc3242481484fedd388b7be098 [id=be02dac770c84af69e2ff172417a074a]) successfully deleted
I20260926 01:55:48.987591 9341 raft_consensus.cc:359] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.994668 9341 raft_consensus.cc:385] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:48.994971 9341 raft_consensus.cc:740] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:48.995672 9341 consensus_queue.cc:260] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:48.996131 9339 tablet_bootstrap.cc:492] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:48.996613 9339 ts_tablet_manager.cc:1403] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.467s user 0.031s sys 0.018s
I20260926 01:55:48.977617 8879 tablet_service.cc:1511] Processing CreateTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 (DEFAULT_TABLE table=default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.998265 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ef5265eb6244aa96c8195aa30cdfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.007688 9341 ts_tablet_manager.cc:1434] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
W20260926 01:55:48.978526 8843 leader_election.cc:343] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:48.980737 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a54eb3b8cf64a478df38df9d4aef02f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:49.009593 9003 leader_election.cc:343] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: Tablet not RUNNING: STOPPED
I20260926 01:55:49.010043 9003 leader_election.cc:304] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:49.011137 9594 raft_consensus.cc:2749] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:49.015683 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 (DEFAULT_TABLE table=default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:48.988979 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 8a56044249bb4aaa81903d2a8aeb008e (DEFAULT_TABLE table=default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.016609 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 (DEFAULT_TABLE table=default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.016909 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ef5265eb6244aa96c8195aa30cdfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.017241 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a56044249bb4aaa81903d2a8aeb008e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.017735 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ef5265eb6244aa96c8195aa30cdfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:48.992551 9648 raft_consensus.cc:493] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:48.992997 9575 raft_consensus.cc:697] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:49.025960 9575 consensus_queue.cc:237] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.026875 9341 tablet_bootstrap.cc:492] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:49.031821 9617 consensus_queue.cc:1048] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:49.035629 9339 raft_consensus.cc:359] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.036343 9339 raft_consensus.cc:385] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.036598 9339 raft_consensus.cc:740] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.037402 9339 consensus_queue.cc:260] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.038842 8878 tablet_service.cc:1511] Processing CreateTablet for tablet f2ee16b0c3e84936abba8248f194786a (DEFAULT_TABLE table=default.Table_2b020c7bf3ca40f5932fadfe0dc566d4 [id=182d678afe7743aba274cc33ba1a445c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.041395 9515 ts_tablet_manager.cc:1929] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:49.042012 9515 log.cc:1199] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/8a011a9edfe94e04afb6ea0367dd979f
I20260926 01:55:49.042884 9515 ts_tablet_manager.cc:1950] T 8a011a9edfe94e04afb6ea0367dd979f P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:49.050825 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 2 got 0
I20260926 01:55:49.053718 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_a4d2c1ed01c74ff3a7260703edd509be [id=2af856781c904c77a11ca9f4966ca44b]). Expected version 3 got 0
I20260926 01:55:49.054968 8531 catalog_manager.cc:5838] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:49.056638 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:49.064193 9341 tablet_bootstrap.cc:654] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.065698 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]) successfully deleted
I20260926 01:55:49.067952 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee16b0c3e84936abba8248f194786a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.070026 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 3571967749844fd7859cf179a93cf403 (DEFAULT_TABLE table=default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.071161 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3571967749844fd7859cf179a93cf403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.034513 9340 tablet_bootstrap.cc:492] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:49.032821 9648 raft_consensus.cc:515] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.079592 9339 ts_tablet_manager.cc:1434] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.083s user 0.007s sys 0.001s
I20260926 01:55:48.984683 9499 raft_consensus.cc:515] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.092114 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 3571967749844fd7859cf179a93cf403 (DEFAULT_TABLE table=default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.093358 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3571967749844fd7859cf179a93cf403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.095422 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 (DEFAULT_TABLE table=default.Table_1ab250b0a79b4325b2c79e44ef3ff928 [id=f29287621a4a4cbea49066dde89fb252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.096526 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2fa5fbe4146b5adb27ff09de878d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.097415 8533 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet cb57d9fd279344bf8245e7fc2cd5e54b
I20260926 01:55:49.097388 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:49.105608 9339 tablet_bootstrap.cc:492] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:49.109156 9047 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f with delete_type TABLET_DATA_DELETED (replaced by tablet 3571967749844fd7859cf179a93cf403) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:49.109551 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:49.110374 9515 tablet_replica.cc:333] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:49.111968 9515 raft_consensus.cc:2243] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260926 01:55:49.112311 8538 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 35) took 798 ms (client timeout 1000 ms). Trace:
I20260926 01:55:49.112428 9515 raft_consensus.cc:2272] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:49.112540 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:49.112649 8538 rpcz_store.cc:269] 0926 01:55:48.313437 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.313563 (+ 126us) service_pool.cc:224] Handling call
0926 01:55:49.112233 (+798670us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":7}
I20260926 01:55:49.113308 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:49.120735 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:49.122926 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b6daaf274894989b90da364f5da5250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:49.124099 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 (DEFAULT_TABLE table=default.Table_1ab250b0a79b4325b2c79e44ef3ff928 [id=f29287621a4a4cbea49066dde89fb252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.125226 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2fa5fbe4146b5adb27ff09de878d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.133281 9515 ts_tablet_manager.cc:1916] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:49.136847 9516 tablet_replica.cc:333] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
W20260926 01:55:49.137220 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet dccbfbc9fb5947f9b85ca8560e205b08 because the tablet was not found. No further retry: Not found: Tablet not found: dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:49.140810 9516 raft_consensus.cc:2243] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:49.144462 9516 pending_rounds.cc:70] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Trying to abort 1 pending ops.
I20260926 01:55:49.150290 9516 pending_rounds.cc:77] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428218284249088 op_type: NO_OP noop_request { }
I20260926 01:55:49.150898 9516 raft_consensus.cc:2889] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:49.152577 9516 raft_consensus.cc:2272] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:49.157640 9516 ts_tablet_manager.cc:1916] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:49.160480 9339 tablet_bootstrap.cc:654] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.173367 9340 tablet_bootstrap.cc:654] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.175158 9648 leader_election.cc:290] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:49.176020 9514 ts_tablet_manager.cc:1929] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260926 01:55:49.175827 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 165819us
I20260926 01:55:49.176364 9514 log.cc:1199] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/8a54eb3b8cf64a478df38df9d4aef02f
I20260926 01:55:49.177042 9514 ts_tablet_manager.cc:1950] T 8a54eb3b8cf64a478df38df9d4aef02f P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:49.177871 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:49.181780 9499 leader_election.cc:290] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:49.191624 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32e0fe60b8744dbb8ae2bafefe3e834d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:49.196593 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 3571967749844fd7859cf179a93cf403 (DEFAULT_TABLE table=default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.197774 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3571967749844fd7859cf179a93cf403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.202903 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 87) took 1987 ms. Trace:
I20260926 01:55:49.203263 9514 rpcz_store.cc:276] 0926 01:55:47.215730 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:47.215915 (+ 185us) service_pool.cc:224] Handling call
0926 01:55:48.816074 (+1600159us) ts_tablet_manager.cc:1151] Deleting tablet 8a54eb3b8cf64a478df38df9d4aef02f
0926 01:55:48.816121 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:49.009219 (+193098us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:49.175886 (+166667us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:49.202836 (+ 26950us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19658,"tablet-delete.queue_time_us":1599546}
I20260926 01:55:49.204620 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 91) took 1737 ms. Trace:
I20260926 01:55:49.204912 9514 rpcz_store.cc:276] 0926 01:55:47.467643 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:47.467792 (+ 149us) service_pool.cc:224] Handling call
0926 01:55:49.204294 (+1736502us) ts_tablet_manager.cc:1151] Deleting tablet f3bb1643da054583b97efd2e1a9e9c2f
0926 01:55:49.204333 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:49.204569 (+ 236us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732455}
I20260926 01:55:49.205864 9514 tablet_replica.cc:333] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:49.206718 9514 raft_consensus.cc:2243] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:49.207710 9514 pending_rounds.cc:70] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Trying to abort 1 pending ops.
I20260926 01:55:49.208060 9514 pending_rounds.cc:77] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333428217120468992 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "a85e47c1bf9f4e96958aa2de41039180" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_8969d22a4ab54a5484838555a31bb684" dest_uuid: "49bd320bf73f4252804ec867ec077510" new_extra_config { } }
I20260926 01:55:49.159860 9653 tablet.cc:1751] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:49.194651 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32e0fe60b8744dbb8ae2bafefe3e834d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:49.222766 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_a8c0c28267ee48f39ee4eb4c11da8a9c" } new_table_name: "default.Table_39a7b6f090bf44b5b0c15825bf0d75e5" modify_external_catalogs: true
I20260926 01:55:49.222373 9218 raft_consensus.cc:2468] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:49.226660 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:49.227285 9219 raft_consensus.cc:3060] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.229928 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_db78756a42ab4ab3924a6b3d285f430b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:49.231603 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db78756a42ab4ab3924a6b3d285f430b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:49.229276 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 (DEFAULT_TABLE table=default.Table_1ab250b0a79b4325b2c79e44ef3ff928 [id=f29287621a4a4cbea49066dde89fb252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:49.235854 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2fa5fbe4146b5adb27ff09de878d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:49.236404 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_fb988b16dc1843a7b22bced528ea5ea6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:49.239200 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_84d6c2cf62bb42779b018cfc68f2b42f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:49.240279 9339 tablet_bootstrap.cc:492] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:49.240733 9339 ts_tablet_manager.cc:1403] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.160s user 0.007s sys 0.003s
W20260926 01:55:49.240880 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84d6c2cf62bb42779b018cfc68f2b42f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:49.238188 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb988b16dc1843a7b22bced528ea5ea6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:49.242378 9219 raft_consensus.cc:2468] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:49.244397 9339 raft_consensus.cc:359] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.245054 9339 raft_consensus.cc:385] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.245296 9339 raft_consensus.cc:740] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.245914 9339 consensus_queue.cc:260] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.256402 9339 ts_tablet_manager.cc:1434] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260926 01:55:49.259807 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 01052a5e22a74a19919bae9dd0296291 from {username='slave'} at 127.0.0.1:42048
W20260926 01:55:49.208863 9514 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58226: Aborted: Op aborted
I20260926 01:55:49.263217 9514 raft_consensus.cc:2272] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:49.267047 9514 ts_tablet_manager.cc:1916] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:49.275089 9654 tablet.cc:1751] T 8dfff96cc29145e7a72213bb5ecca1ae P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:49.277537 9659 tablet.cc:1751] T 8dfff96cc29145e7a72213bb5ecca1ae P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:49.279371 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 8a54eb3b8cf64a478df38df9d4aef02f (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]) successfully deleted
W20260926 01:55:49.280138 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet f3bb1643da054583b97efd2e1a9e9c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bb1643da054583b97efd2e1a9e9c2f already in progress: creating tablet
I20260926 01:55:49.282132 8540 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet c36ea09cf15647c0a344c9ba9b289d61
I20260926 01:55:49.286429 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:49.297410 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:49.299130 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:49.286614 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:49.287635 9339 tablet_bootstrap.cc:492] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:55:49.321831 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 943a990fa10948e8961c02e9999569f5 (table default.Table_3907f5fafff14aec94d0a9e331e86dc9 [id=cc59eec581574b65a6332863423dab90], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 943a990fa10948e8961c02e9999569f5 has no consensus state
I20260926 01:55:49.260907 9515 ts_tablet_manager.cc:1929] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:49.322661 9515 log.cc:1199] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/8a54eb3b8cf64a478df38df9d4aef02f
I20260926 01:55:49.323388 9515 ts_tablet_manager.cc:1950] T 8a54eb3b8cf64a478df38df9d4aef02f P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:49.326267 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 8a54eb3b8cf64a478df38df9d4aef02f (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]) successfully deleted
I20260926 01:55:49.327307 9515 tablet_replica.cc:333] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:49.328126 9515 raft_consensus.cc:2243] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:49.328670 9515 raft_consensus.cc:2272] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:49.330513 9515 ts_tablet_manager.cc:1916] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:49.338759 8536 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 1cba678b66744dcfb4d13ae07ebec33f
I20260926 01:55:49.340260 8536 catalog_manager.cc:6934] ReplaceTablet: tablet 1cba678b66744dcfb4d13ae07ebec33f deleted and replaced by tablet 880d77d42c394e6a9cf0b3768a915402
I20260926 01:55:49.345564 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:49.336578 9339 tablet_bootstrap.cc:654] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.346745 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:49.348110 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 2 got 0
I20260926 01:55:49.348765 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_a4d2c1ed01c74ff3a7260703edd509be [id=2af856781c904c77a11ca9f4966ca44b]). Expected version 3 got 0
I20260926 01:55:49.358175 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:49.279444 8843 leader_election.cc:343] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:49.298405 9514 ts_tablet_manager.cc:1929] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:49.338496 9340 tablet_bootstrap.cc:492] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
W20260926 01:55:49.362740 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 83331.4us
I20260926 01:55:49.363394 9340 ts_tablet_manager.cc:1403] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.329s user 0.008s sys 0.006s
W20260926 01:55:49.363947 8843 leader_election.cc:343] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:49.366389 8524 catalog_manager.cc:5142] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): alter failed for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]): Aborted: Op aborted
I20260926 01:55:49.362782 9514 log.cc:1199] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/a85e47c1bf9f4e96958aa2de41039180
I20260926 01:55:49.369926 9514 ts_tablet_manager.cc:1950] T a85e47c1bf9f4e96958aa2de41039180 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:49.372229 9617 consensus_queue.cc:1048] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:49.376729 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a54eb3b8cf64a478df38df9d4aef02f with delete_type TABLET_DATA_DELETED (replaced by tablet 3571967749844fd7859cf179a93cf403) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:49.376995 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet 9d10be686d9045efaf384d326b356e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:49.377274 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 3695411d251847b4b1ab0a1eb936a55a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:43 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:49.377353 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:49.358111 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_cf700d4a4dcb4da1afbc424ea51d6f77" } modify_external_catalogs: true
I20260926 01:55:49.378124 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_cf700d4a4dcb4da1afbc424ea51d6f77" } modify_external_catalogs: true
I20260926 01:55:49.380265 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 92) took 1786 ms. Trace:
I20260926 01:55:49.380667 9514 rpcz_store.cc:276] 0926 01:55:47.593798 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:47.593920 (+ 122us) service_pool.cc:224] Handling call
0926 01:55:49.205722 (+1611802us) ts_tablet_manager.cc:1151] Deleting tablet a85e47c1bf9f4e96958aa2de41039180
0926 01:55:49.205760 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:49.274445 (+ 68685us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:49.298277 (+ 23832us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:49.380200 (+ 81923us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6464,"tablet-delete.queue_time_us":1611073}
I20260926 01:55:49.384987 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 94) took 1115 ms. Trace:
I20260926 01:55:49.385442 9514 rpcz_store.cc:276] 0926 01:55:48.269310 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.277779 (+ 8469us) service_pool.cc:224] Handling call
0926 01:55:49.382447 (+1104668us) ts_tablet_manager.cc:1151] Deleting tablet 099af935249a4310b29852a258661794
0926 01:55:49.384559 (+ 2112us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:49.384905 (+ 346us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104027}
I20260926 01:55:49.386610 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 95) took 1107 ms. Trace:
I20260926 01:55:49.386950 9514 rpcz_store.cc:276] 0926 01:55:48.279248 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.282835 (+ 3587us) service_pool.cc:224] Handling call
0926 01:55:49.386349 (+1103514us) ts_tablet_manager.cc:1151] Deleting tablet 28bd8695a6b34585bdb1bdabf1bcc654
0926 01:55:49.386388 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:49.386558 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1102936}
I20260926 01:55:49.388020 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 96) took 1100 ms. Trace:
I20260926 01:55:49.388449 9514 rpcz_store.cc:276] 0926 01:55:48.287223 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.298413 (+ 11190us) service_pool.cc:224] Handling call
0926 01:55:49.387787 (+1089374us) ts_tablet_manager.cc:1151] Deleting tablet 9d9cb4de16da44e1b90ac79e9e32cb88
0926 01:55:49.387826 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:49.387971 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1088725}
W20260926 01:55:49.388742 8531 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 35) took 1079 ms (client timeout 1000 ms). Trace:
I20260926 01:55:49.392585 9339 tablet_bootstrap.cc:492] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:49.393105 9339 ts_tablet_manager.cc:1403] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.106s user 0.011s sys 0.000s
I20260926 01:55:49.396788 9339 raft_consensus.cc:359] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.397487 9339 raft_consensus.cc:385] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.397750 9339 raft_consensus.cc:740] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.398942 9514 tablet_replica.cc:333] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:49.399835 9514 raft_consensus.cc:2243] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:49.400382 9514 raft_consensus.cc:2272] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:49.397032 9340 raft_consensus.cc:359] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.402575 9340 raft_consensus.cc:385] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.402915 9340 raft_consensus.cc:740] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:55:49.389093 8531 rpcz_store.cc:269] 0926 01:55:48.309822 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.310960 (+ 1138us) service_pool.cc:224] Handling call
0926 01:55:49.388658 (+1077698us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:49.305059 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:49.305473 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:49.305499 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:49.305512 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:49.305782 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:49.305822 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:49.305846 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:49.305996 (+ 150us) tablet.cc:701] Row locks acquired
0926 01:55:49.306009 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:49.306140 (+ 131us) write_op.cc:270] Start()
0926 01:55:49.306230 (+ 90us) write_op.cc:276] Timestamp: P: 1790387749306055 usec, L: 0
0926 01:55:49.306245 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:49.334032 (+ 27787us) log.cc:844] Serialized 530 byte log entry
0926 01:55:49.342777 (+ 8745us) op_driver.cc:464] REPLICATION: finished
0926 01:55:49.343089 (+ 312us) write_op.cc:301] APPLY: starting
0926 01:55:49.343142 (+ 53us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:49.343237 (+ 95us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:49.343339 (+ 102us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:49.343498 (+ 159us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:49.343650 (+ 152us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:49.343689 (+ 39us) write_op.cc:312] APPLY: finished
0926 01:55:49.343886 (+ 197us) log.cc:844] Serialized 23 byte log entry
0926 01:55:49.344036 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:49.344095 (+ 59us) write_op.cc:454] Released schema lock
0926 01:55:49.344131 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":1227,"apply.run_wall_time_us":1227,"num_ops":1,"prepare.queue_time_us":205665,"replication_time_us":36440}]]}
I20260926 01:55:49.403879 9340 consensus_queue.cc:260] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.398130 8878 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:49.391592 8843 leader_election.cc:304] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
W20260926 01:55:49.406970 8843 raft_consensus.cc:2686] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 01:55:49.408037 8843 leader_election.cc:304] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:49.409791 9617 raft_consensus.cc:2804] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:49.410710 9575 consensus_queue.cc:1048] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:49.410239 9617 raft_consensus.cc:697] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:49.412520 9617 consensus_queue.cc:237] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.418784 9339 consensus_queue.cc:260] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.424304 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]) successfully deleted
W20260926 01:55:49.425534 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 099af935249a4310b29852a258661794 because the tablet was not found. No further retry: Not found: Tablet not found: 099af935249a4310b29852a258661794
W20260926 01:55:49.427737 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 28bd8695a6b34585bdb1bdabf1bcc654 because the tablet was not found. No further retry: Not found: Tablet not found: 28bd8695a6b34585bdb1bdabf1bcc654
W20260926 01:55:49.429709 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 9d9cb4de16da44e1b90ac79e9e32cb88 because the tablet was not found. No further retry: Not found: Tablet not found: 9d9cb4de16da44e1b90ac79e9e32cb88
I20260926 01:55:49.430078 9653 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:49.445785 9667 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:49.448868 9339 ts_tablet_manager.cc:1434] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.055s user 0.003s sys 0.005s
I20260926 01:55:49.449815 9341 tablet_bootstrap.cc:492] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:49.450290 9341 ts_tablet_manager.cc:1403] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.424s user 0.010s sys 0.001s
I20260926 01:55:49.453173 9340 ts_tablet_manager.cc:1434] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.089s user 0.002s sys 0.004s
I20260926 01:55:49.459475 9339 tablet_bootstrap.cc:492] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:49.481298 8878 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:49.482674 9515 ts_tablet_manager.cc:1929] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:49.483134 9515 log.cc:1199] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:49.483862 9515 ts_tablet_manager.cc:1950] T dccbfbc9fb5947f9b85ca8560e205b08 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:49.513557 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet dccbfbc9fb5947f9b85ca8560e205b08 (table default.Table_14869892bcf7429385632e2df181743c [id=bfa8586ef17949bead9731402cd78042]) successfully deleted
I20260926 01:55:49.523952 9339 tablet_bootstrap.cc:654] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:49.531663 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 3695411d251847b4b1ab0a1eb936a55a because the tablet was not found. No further retry: Not found: Tablet not found: 3695411d251847b4b1ab0a1eb936a55a
W20260926 01:55:49.533730 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet c36ea09cf15647c0a344c9ba9b289d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c36ea09cf15647c0a344c9ba9b289d61 already in progress: creating tablet
W20260926 01:55:49.535584 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 1cba678b66744dcfb4d13ae07ebec33f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cba678b66744dcfb4d13ae07ebec33f already in progress: creating tablet
I20260926 01:55:49.536537 9515 tablet_replica.cc:333] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:49.537428 9515 raft_consensus.cc:2243] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:49.537962 9515 raft_consensus.cc:2272] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:49.539853 9515 ts_tablet_manager.cc:1916] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:49.518168 8843 consensus_peers.cc:597] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a011a9edfe94e04afb6ea0367dd979f. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:49.536937 9514 ts_tablet_manager.cc:1916] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:49.541296 9340 tablet_bootstrap.cc:492] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:49.540715 9594 raft_consensus.cc:493] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:49.557145 9594 raft_consensus.cc:515] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.559305 9594 leader_election.cc:290] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:49.561460 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:49.563199 9670 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:49.564436 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:49.567255 9006 leader_election.cc:304] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:49.568483 9594 raft_consensus.cc:2749] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:49.600699 9339 tablet_bootstrap.cc:492] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:49.601296 9339 ts_tablet_manager.cc:1403] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.142s user 0.007s sys 0.005s
I20260926 01:55:49.604838 9339 raft_consensus.cc:359] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.609491 9339 raft_consensus.cc:385] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.609833 9339 raft_consensus.cc:740] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.616003 8534 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_1ab250b0a79b4325b2c79e44ef3ff928" } new_table_name: "default.Table_ae4f6e78de7a4f6c8f96a3b179cf7c6d" modify_external_catalogs: true
I20260926 01:55:49.610710 9339 consensus_queue.cc:260] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.620425 9339 ts_tablet_manager.cc:1434] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260926 01:55:49.625459 9339 tablet_bootstrap.cc:492] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:49.648962 9339 tablet_bootstrap.cc:654] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.653921 9072 raft_consensus.cc:3060] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.661339 9515 ts_tablet_manager.cc:1929] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:49.661860 9515 log.cc:1199] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/1866cd97bb024a9baecf101a6d5b1ee4
W20260926 01:55:49.677531 9195 consensus_meta.cc:287] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.517s user 0.006s sys 0.006s
I20260926 01:55:49.682744 9515 ts_tablet_manager.cc:1950] T 1866cd97bb024a9baecf101a6d5b1ee4 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:49.690896 9341 raft_consensus.cc:359] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.691596 9341 raft_consensus.cc:385] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.691888 9341 raft_consensus.cc:740] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.692546 9341 consensus_queue.cc:260] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.697851 9341 ts_tablet_manager.cc:1434] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.247s user 0.007s sys 0.000s
I20260926 01:55:49.705013 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 1866cd97bb024a9baecf101a6d5b1ee4 (table default.Table_8e5e15ce8e58425697a7f013c09c0c72 [id=5dd626e9121c4d5aba51637d7c6cb8a2]) successfully deleted
I20260926 01:55:49.710135 8532 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_92899e57b6c14f62b2a999bb2487f558" } modify_external_catalogs: true
I20260926 01:55:49.710839 8532 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_92899e57b6c14f62b2a999bb2487f558" } modify_external_catalogs: true
I20260926 01:55:49.714107 9617 consensus_queue.cc:1048] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:49.725574 9339 tablet_bootstrap.cc:492] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:49.726044 9339 ts_tablet_manager.cc:1403] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.101s user 0.010s sys 0.002s
I20260926 01:55:49.742980 9339 raft_consensus.cc:359] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.743703 9339 raft_consensus.cc:385] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.743980 9339 raft_consensus.cc:740] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.744714 9339 consensus_queue.cc:260] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.749403 9339 ts_tablet_manager.cc:1434] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260926 01:55:49.750310 9339 tablet_bootstrap.cc:492] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:49.764654 9339 tablet_bootstrap.cc:654] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.766932 8537 master_service.cc:960] ReplaceTablet: received request to replace tablet 23d2805b41d645d3bb905f26a380f383 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:49.782332 9340 tablet_bootstrap.cc:654] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.784299 9339 tablet_bootstrap.cc:492] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:49.784746 9339 ts_tablet_manager.cc:1403] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.003s
I20260926 01:55:49.787261 9339 raft_consensus.cc:359] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.787920 9339 raft_consensus.cc:385] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.788159 9339 raft_consensus.cc:740] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.788882 9339 consensus_queue.cc:260] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.791710 9341 tablet_bootstrap.cc:492] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:49.793574 9339 ts_tablet_manager.cc:1434] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 01:55:49.796607 9339 tablet_bootstrap.cc:492] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:49.814282 9516 ts_tablet_manager.cc:1929] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:49.814971 9516 log.cc:1199] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/1866cd97bb024a9baecf101a6d5b1ee4
I20260926 01:55:49.815865 9516 ts_tablet_manager.cc:1950] T 1866cd97bb024a9baecf101a6d5b1ee4 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:49.802347 9339 tablet_bootstrap.cc:654] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.825390 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_f820adee11bf4557b6119642992da842"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:49.827121 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f820adee11bf4557b6119642992da842 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:49.844762 9339 tablet_bootstrap.cc:492] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:49.845309 9339 ts_tablet_manager.cc:1403] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.003s
I20260926 01:55:49.848112 9339 raft_consensus.cc:359] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.848807 9339 raft_consensus.cc:385] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.849049 9339 raft_consensus.cc:740] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.849869 9339 consensus_queue.cc:260] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.851869 9339 ts_tablet_manager.cc:1434] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260926 01:55:49.857426 9339 tablet_bootstrap.cc:492] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:49.863204 9339 tablet_bootstrap.cc:654] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.871744 9523 raft_consensus.cc:493] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:49.872264 9523 raft_consensus.cc:515] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.876353 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01052a5e22a74a19919bae9dd0296291" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:49.876349 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01052a5e22a74a19919bae9dd0296291" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:49.877030 9072 raft_consensus.cc:2468] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:49.877027 8906 raft_consensus.cc:2468] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:49.874362 9523 leader_election.cc:290] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:49.878432 9152 leader_election.cc:304] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:49.879683 9523 raft_consensus.cc:2804] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:49.880101 9523 raft_consensus.cc:493] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:49.880501 9523 raft_consensus.cc:3060] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.891245 8538 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet fe283c77306e4623a70e020577b37227
I20260926 01:55:49.893953 9523 raft_consensus.cc:515] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.898281 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01052a5e22a74a19919bae9dd0296291" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:49.898497 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01052a5e22a74a19919bae9dd0296291" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:49.898968 9072 raft_consensus.cc:3060] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.899051 8906 raft_consensus.cc:3060] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.907081 8906 raft_consensus.cc:2468] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:49.908499 9152 leader_election.cc:304] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:49.909232 9072 raft_consensus.cc:2468] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:49.909384 9635 raft_consensus.cc:2804] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:49.909832 9635 raft_consensus.cc:697] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:49.910698 9635 consensus_queue.cc:237] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:49.912899 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_129e4100f7f245e9877924fd11902aea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:49.914788 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_129e4100f7f245e9877924fd11902aea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:49.917176 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 1866cd97bb024a9baecf101a6d5b1ee4 (table default.Table_8e5e15ce8e58425697a7f013c09c0c72 [id=5dd626e9121c4d5aba51637d7c6cb8a2]) successfully deleted
I20260926 01:55:49.919488 9523 leader_election.cc:290] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:49.920099 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 3b6daaf274894989b90da364f5da5250 because the tablet was not found. No further retry: Not found: Tablet not found: 3b6daaf274894989b90da364f5da5250
I20260926 01:55:49.923316 9341 tablet_bootstrap.cc:654] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.925395 9339 tablet_bootstrap.cc:492] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:49.925856 9339 ts_tablet_manager.cc:1403] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.016s
I20260926 01:55:49.927652 9594 raft_consensus.cc:493] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:49.928133 9594 raft_consensus.cc:515] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.929288 9339 raft_consensus.cc:359] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.930109 9339 raft_consensus.cc:385] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.931455 9594 leader_election.cc:290] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:49.931519 9339 raft_consensus.cc:740] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
W20260926 01:55:49.924909 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet c36ea09cf15647c0a344c9ba9b289d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c36ea09cf15647c0a344c9ba9b289d61 already in progress: creating tablet
I20260926 01:55:49.932395 9339 consensus_queue.cc:260] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.932950 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d101ad03f8e4d7f8bf70ea993c4f58b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:49.932899 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d101ad03f8e4d7f8bf70ea993c4f58b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:49.933547 9218 raft_consensus.cc:2468] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:49.935398 9006 leader_election.cc:304] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:55:49.936616 9003 leader_election.cc:343] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:49.940438 9594 raft_consensus.cc:2804] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:49.940815 9594 raft_consensus.cc:493] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:49.941130 9594 raft_consensus.cc:3060] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.951753 9339 ts_tablet_manager.cc:1434] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260926 01:55:49.952850 9339 tablet_bootstrap.cc:492] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:49.965341 9340 tablet_bootstrap.cc:492] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:49.965826 9340 ts_tablet_manager.cc:1403] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.425s user 0.014s sys 0.001s
I20260926 01:55:49.963022 9594 raft_consensus.cc:515] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.968344 9340 raft_consensus.cc:359] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.968968 9340 raft_consensus.cc:385] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:49.969216 9340 raft_consensus.cc:740] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:49.975849 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d101ad03f8e4d7f8bf70ea993c4f58b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:49.976440 9218 raft_consensus.cc:3060] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.980319 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d101ad03f8e4d7f8bf70ea993c4f58b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:49.982091 9070 raft_consensus.cc:3060] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:49.968344 9594 leader_election.cc:290] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:49.984812 9339 tablet_bootstrap.cc:654] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:49.987286 9218 raft_consensus.cc:2468] T 6d101ad03f8e4d7f8bf70ea993c4f58b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
W20260926 01:55:49.987576 9003 leader_election.cc:343] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:49.988427 9006 leader_election.cc:304] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:49.989511 9594 raft_consensus.cc:2804] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:55:49.989624 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 1cba678b66744dcfb4d13ae07ebec33f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cba678b66744dcfb4d13ae07ebec33f already in progress: creating tablet
I20260926 01:55:49.989964 9594 raft_consensus.cc:697] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:49.990839 9594 consensus_queue.cc:237] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:49.997044 9617 consensus_queue.cc:1048] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:55:50.000854 8534 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bf2fa5fbe4146b5adb27ff09de878d3 consensus state has no leader
I20260926 01:55:50.008383 9683 raft_consensus.cc:493] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.008893 9683 raft_consensus.cc:515] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:50.017750 9683 leader_election.cc:290] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.019764 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfaca3fc02634f459083d23f3102f725" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:50.024103 9003 leader_election.cc:343] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:50.024737 9006 leader_election.cc:343] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: dfaca3fc02634f459083d23f3102f725
I20260926 01:55:50.025190 9006 leader_election.cc:304] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:50.026693 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_a2babbbea5d2444d8747712d56064903"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:50.027740 9683 raft_consensus.cc:2749] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:55:50.028599 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2babbbea5d2444d8747712d56064903 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:50.037214 9659 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260926 01:55:50.046312 9680 log.cc:927] Time spent T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260926 01:55:49.854234 9514 ts_tablet_manager.cc:1929] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:50.051211 9514 log.cc:1199] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:50.053244 9514 ts_tablet_manager.cc:1950] T 32e0fe60b8744dbb8ae2bafefe3e834d P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:50.054919 9341 tablet_bootstrap.cc:492] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:50.055347 9341 ts_tablet_manager.cc:1403] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.264s user 0.007s sys 0.003s
I20260926 01:55:50.058287 9689 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:50.063683 9654 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:50.097532 9617 consensus_queue.cc:1048] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:50.135028 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_632a5e0f37424fc9a71a1b75d1a4814c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:50.137305 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_632a5e0f37424fc9a71a1b75d1a4814c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:50.062297 9341 raft_consensus.cc:359] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.187150 9341 raft_consensus.cc:385] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.187486 9341 raft_consensus.cc:740] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.188191 9341 consensus_queue.cc:260] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.190235 9341 ts_tablet_manager.cc:1434] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.135s user 0.003s sys 0.003s
I20260926 01:55:50.192991 9341 tablet_bootstrap.cc:492] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:50.199640 9340 consensus_queue.cc:260] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.206375 9341 tablet_bootstrap.cc:654] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.227037 9341 tablet_bootstrap.cc:492] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:50.227519 9341 ts_tablet_manager.cc:1403] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.003s
I20260926 01:55:50.230413 9341 raft_consensus.cc:359] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.231105 9341 raft_consensus.cc:385] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.231825 9341 raft_consensus.cc:740] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.232620 9341 consensus_queue.cc:260] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.239225 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 98) took 1792 ms. Trace:
I20260926 01:55:50.239624 9514 rpcz_store.cc:276] 0926 01:55:48.447199 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.447852 (+ 653us) service_pool.cc:224] Handling call
0926 01:55:49.398759 (+950907us) ts_tablet_manager.cc:1151] Deleting tablet 32e0fe60b8744dbb8ae2bafefe3e834d
0926 01:55:49.398819 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:49.735356 (+336537us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:49.854083 (+118727us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:50.239138 (+385055us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":950153}
I20260926 01:55:50.241596 9341 ts_tablet_manager.cc:1434] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.014s user 0.001s sys 0.005s
I20260926 01:55:50.244086 9341 tablet_bootstrap.cc:492] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:50.250351 9339 tablet_bootstrap.cc:492] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:50.250855 9339 ts_tablet_manager.cc:1403] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.298s user 0.036s sys 0.084s
I20260926 01:55:50.254231 9339 raft_consensus.cc:359] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.254935 9339 raft_consensus.cc:385] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.255189 9339 raft_consensus.cc:740] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.255942 9339 consensus_queue.cc:260] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.257900 9339 ts_tablet_manager.cc:1434] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:55:50.258919 9341 tablet_bootstrap.cc:654] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.260527 9339 tablet_bootstrap.cc:492] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:50.264294 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 100) took 1659 ms. Trace:
I20260926 01:55:50.264650 9514 rpcz_store.cc:276] 0926 01:55:48.605203 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.605314 (+ 111us) service_pool.cc:224] Handling call
0926 01:55:50.263886 (+1658572us) ts_tablet_manager.cc:1151] Deleting tablet dfaca3fc02634f459083d23f3102f725
0926 01:55:50.263937 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.264229 (+ 292us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1657944}
I20260926 01:55:50.267732 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 103) took 1566 ms. Trace:
W20260926 01:55:50.266111 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet dfaca3fc02634f459083d23f3102f725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfaca3fc02634f459083d23f3102f725 already in progress: creating tablet
I20260926 01:55:50.285961 9341 tablet_bootstrap.cc:492] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:50.287000 9341 ts_tablet_manager.cc:1403] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.011s
I20260926 01:55:50.289510 9683 raft_consensus.cc:493] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.289518 9341 raft_consensus.cc:359] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.290155 9341 raft_consensus.cc:385] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.290000 9683 raft_consensus.cc:515] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.290385 9341 raft_consensus.cc:740] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.291030 9341 consensus_queue.cc:260] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.291986 9683 leader_election.cc:290] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:50.292970 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1df1d4daaddc4ac4aa02aa7763ea3184" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:50.294548 9003 leader_election.cc:343] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:50.299782 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1df1d4daaddc4ac4aa02aa7763ea3184" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:55:50.302276 9006 leader_election.cc:343] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:50.302734 9006 leader_election.cc:304] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:50.304320 9683 raft_consensus.cc:2749] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:50.203125 9340 ts_tablet_manager.cc:1434] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.237s user 0.007s sys 0.001s
I20260926 01:55:50.306900 9341 ts_tablet_manager.cc:1434] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20260926 01:55:50.307400 9340 tablet_bootstrap.cc:492] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:50.274562 9514 rpcz_store.cc:276] 0926 01:55:48.701767 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.702871 (+ 1104us) service_pool.cc:224] Handling call
0926 01:55:50.267035 (+1564164us) ts_tablet_manager.cc:1151] Deleting tablet 3b6daaf274894989b90da364f5da5250
0926 01:55:50.267141 (+ 106us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.267643 (+ 502us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563410}
I20260926 01:55:50.321413 9514 tablet_replica.cc:333] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
W20260926 01:55:50.321715 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 3b6daaf274894989b90da364f5da5250 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b6daaf274894989b90da364f5da5250 already in progress: creating tablet
I20260926 01:55:50.322190 9514 raft_consensus.cc:2243] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:50.324031 9341 tablet_bootstrap.cc:492] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:50.330371 9341 tablet_bootstrap.cc:654] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.339314 9339 tablet_bootstrap.cc:654] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.343075 9514 raft_consensus.cc:2272] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:50.344939 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 (DEFAULT_TABLE table=default.Table_87c331af29e7419593789c486dc50a13 [id=2fa9e93da5704e3b9970e16f5afe93bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.346292 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f666b1c069435b8ff0a3fae9f1e873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.346395 9514 ts_tablet_manager.cc:1916] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:50.363708 9340 tablet_bootstrap.cc:654] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.367457 9523 raft_consensus.cc:493] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.367959 9523 raft_consensus.cc:515] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.370211 9523 leader_election.cc:290] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.375883 9341 tablet_bootstrap.cc:492] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:50.376407 9341 ts_tablet_manager.cc:1403] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.053s user 0.020s sys 0.007s
I20260926 01:55:50.376317 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75d4598596ff49378fc3536fff0ac1b1" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:50.379103 9341 raft_consensus.cc:359] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.334084 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 01052a5e22a74a19919bae9dd0296291 (table default.Table_a8c0c28267ee48f39ee4eb4c11da8a9c [id=6a3cae01c8664f3d8d4b89e8a91665f7]). Expected version 2 got 0
W20260926 01:55:50.379511 9152 leader_election.cc:343] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:50.383870 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 (DEFAULT_TABLE table=default.Table_87c331af29e7419593789c486dc50a13 [id=2fa9e93da5704e3b9970e16f5afe93bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.385128 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f666b1c069435b8ff0a3fae9f1e873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.388981 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75d4598596ff49378fc3536fff0ac1b1" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:50.390172 9654 tablet.cc:1746] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Skipping requested alter to schema version 2, tablet already version 2
I20260926 01:55:50.391448 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_a4d2c1ed01c74ff3a7260703edd509be [id=2af856781c904c77a11ca9f4966ca44b]). Expected version 3 got 0
I20260926 01:55:50.392329 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 2 got 0
W20260926 01:55:50.392947 9155 leader_election.cc:343] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:50.393417 9683 raft_consensus.cc:493] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.393392 9155 leader_election.cc:304] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:50.393886 9683 raft_consensus.cc:515] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:50.394766 9635 raft_consensus.cc:2749] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:50.395773 9340 tablet_bootstrap.cc:492] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:50.395905 9683 leader_election.cc:290] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.396255 9340 ts_tablet_manager.cc:1403] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.000s
I20260926 01:55:50.341252 8531 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
W20260926 01:55:50.399093 9003 leader_election.cc:343] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:50.400450 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32e0fe60b8744dbb8ae2bafefe3e834d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:50.401078 9218 raft_consensus.cc:2468] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:50.402426 9006 leader_election.cc:304] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:50.403710 9683 raft_consensus.cc:2804] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:50.404480 8536 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 2 got 0
I20260926 01:55:50.405156 8536 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204]). Expected version 3 got 2
I20260926 01:55:50.404373 9594 raft_consensus.cc:493] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.405654 9594 raft_consensus.cc:515] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:50.406189 8536 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_a4d2c1ed01c74ff3a7260703edd509be [id=2af856781c904c77a11ca9f4966ca44b]). Expected version 3 got 0
I20260926 01:55:50.407724 9594 leader_election.cc:290] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.409418 9340 raft_consensus.cc:359] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:50.409821 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:50.410048 9340 raft_consensus.cc:385] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.413255 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:50.414057 9218 raft_consensus.cc:2393] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:55:50.414883 9683 raft_consensus.cc:493] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:50.415354 9683 raft_consensus.cc:3060] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:50.416360 9006 leader_election.cc:304] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:50.419684 9594 raft_consensus.cc:3060] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:50.410319 9340 raft_consensus.cc:740] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.424562 9617 consensus_queue.cc:1048] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:50.425354 9340 consensus_queue.cc:260] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:50.349838 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 (table default.Table_ae4f6e78de7a4f6c8f96a3b179cf7c6d [id=f29287621a4a4cbea49066dde89fb252], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3bf2fa5fbe4146b5adb27ff09de878d3 consensus state has no leader
W20260926 01:55:50.407053 8531 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 37) took 1031 ms (client timeout 1000 ms). Trace:
I20260926 01:55:50.433650 9659 tablet.cc:1746] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Skipping requested alter to schema version 2, tablet already version 2
W20260926 01:55:50.433892 8531 rpcz_store.cc:269] 0926 01:55:49.375543 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.407957 (+ 32414us) service_pool.cc:224] Handling call
0926 01:55:50.406964 (+999007us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260926 01:55:50.436091 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 9bac55da5b164c0f981914d39c87c33e (DEFAULT_TABLE table=default.Table_3478e66c0ff74bc4a27eb418e346e4e2 [id=05847c8c0e744754b5456a6773f5e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.436714 9340 ts_tablet_manager.cc:1434] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260926 01:55:50.437291 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bac55da5b164c0f981914d39c87c33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.444801 9340 tablet_bootstrap.cc:492] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:50.452718 9594 raft_consensus.cc:2749] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:50.382074 9341 raft_consensus.cc:385] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.453871 9341 raft_consensus.cc:740] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.454631 9341 consensus_queue.cc:260] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.457674 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 9bac55da5b164c0f981914d39c87c33e (DEFAULT_TABLE table=default.Table_3478e66c0ff74bc4a27eb418e346e4e2 [id=05847c8c0e744754b5456a6773f5e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.459183 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bac55da5b164c0f981914d39c87c33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.458597 8536 catalog_manager.cc:5838] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:50.460578 8536 catalog_manager.cc:5838] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:50.466266 9689 tablet.cc:1746] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Skipping requested alter to schema version 2, tablet already version 2
I20260926 01:55:50.452752 9514 ts_tablet_manager.cc:1929] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:50.468888 9514 log.cc:1199] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/8a011a9edfe94e04afb6ea0367dd979f
I20260926 01:55:50.469669 9514 ts_tablet_manager.cc:1950] T 8a011a9edfe94e04afb6ea0367dd979f P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:50.471668 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 229f4cdd841949e4a12e8555cdfe42cc (DEFAULT_TABLE table=default.Table_6f6a2a6f414644f7b5d7a322965922fa [id=e7f06f7838a740a09da1cfda5d27fc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.472735 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229f4cdd841949e4a12e8555cdfe42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.475661 9341 ts_tablet_manager.cc:1434] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.099s user 0.005s sys 0.000s
I20260926 01:55:50.482412 9683 raft_consensus.cc:515] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:50.484319 9340 tablet_bootstrap.cc:654] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.485195 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32e0fe60b8744dbb8ae2bafefe3e834d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:50.485733 9219 raft_consensus.cc:3060] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:50.490139 9683 leader_election.cc:290] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.493721 9219 raft_consensus.cc:2468] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:50.496984 9006 leader_election.cc:304] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:55:50.498436 9003 leader_election.cc:343] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:50.497925 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 104) took 1787 ms. Trace:
I20260926 01:55:50.506742 9523 raft_consensus.cc:493] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.507253 9523 raft_consensus.cc:515] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.505174 9514 rpcz_store.cc:276] 0926 01:55:48.710610 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:48.712730 (+ 2120us) service_pool.cc:224] Handling call
0926 01:55:50.321247 (+1608517us) ts_tablet_manager.cc:1151] Deleting tablet 8a011a9edfe94e04afb6ea0367dd979f
0926 01:55:50.321294 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.406277 (+ 84983us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:50.452617 (+ 46340us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:50.497858 (+ 45241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6528,"tablet-delete.queue_time_us":1607846}
I20260926 01:55:50.509213 9523 leader_election.cc:290] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.510056 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23d2805b41d645d3bb905f26a380f383" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:50.511341 8878 tablet_service.cc:1511] Processing CreateTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 (DEFAULT_TABLE table=default.Table_87c331af29e7419593789c486dc50a13 [id=2fa9e93da5704e3b9970e16f5afe93bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:50.511547 9155 leader_election.cc:343] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:50.513545 9617 consensus_queue.cc:1048] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:50.513592 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 9bac55da5b164c0f981914d39c87c33e (DEFAULT_TABLE table=default.Table_3478e66c0ff74bc4a27eb418e346e4e2 [id=05847c8c0e744754b5456a6773f5e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.519817 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f666b1c069435b8ff0a3fae9f1e873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.522498 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bac55da5b164c0f981914d39c87c33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.523634 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23d2805b41d645d3bb905f26a380f383" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:50.526069 9340 tablet_bootstrap.cc:492] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:50.526499 9340 ts_tablet_manager.cc:1403] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.003s
I20260926 01:55:50.526484 9594 raft_consensus.cc:2804] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:50.528753 9341 tablet_bootstrap.cc:492] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:50.529754 9152 leader_election.cc:343] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:50.530225 9152 leader_election.cc:304] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:50.531018 9523 raft_consensus.cc:2749] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:50.538902 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 8a011a9edfe94e04afb6ea0367dd979f (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]) successfully deleted
W20260926 01:55:50.539928 8524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20260926 01:55:50.544569 9340 raft_consensus.cc:359] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.545275 9340 raft_consensus.cc:385] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.545533 9340 raft_consensus.cc:740] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.548476 9594 raft_consensus.cc:697] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:50.548492 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 114) took 1172 ms. Trace:
I20260926 01:55:50.552050 9340 consensus_queue.cc:260] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.553943 9514 rpcz_store.cc:276] 0926 01:55:49.376528 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.376618 (+ 90us) service_pool.cc:224] Handling call
0926 01:55:50.508791 (+1132173us) ts_tablet_manager.cc:1151] Deleting tablet 8a54eb3b8cf64a478df38df9d4aef02f
0926 01:55:50.548176 (+ 39385us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b5800000f98. stack: 00007f94aec19323 00007f94aec19220 00007f94ae7bd574 0000000000516964 00000000005168f9 00007f94aeacf8a1 00007f94aec13b51 00007f94aec14182 00007f94b7b872c9 00007f94b6d98918 00007f94b6e5f2c9 00007f94b6e5f8c3 00007f94b6e90960 00007f94b6e76e6d 00007f94b6e76e19 00007f94b6e76da9
0926 01:55:50.548227 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.548425 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68144384,"tablet-delete.queue_time_us":1131624}
I20260926 01:55:50.558346 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 115) took 1181 ms. Trace:
I20260926 01:55:50.559068 9514 rpcz_store.cc:276] 0926 01:55:49.376765 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.376871 (+ 106us) service_pool.cc:224] Handling call
0926 01:55:50.558071 (+1181200us) ts_tablet_manager.cc:1151] Deleting tablet 9d10be686d9045efaf384d326b356e0e
0926 01:55:50.558116 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.558286 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180706}
I20260926 01:55:50.560266 9340 ts_tablet_manager.cc:1434] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20260926 01:55:50.560793 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 116) took 1183 ms. Trace:
I20260926 01:55:50.567112 9514 rpcz_store.cc:276] 0926 01:55:49.377056 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.377152 (+ 96us) service_pool.cc:224] Handling call
0926 01:55:50.560537 (+1183385us) ts_tablet_manager.cc:1151] Deleting tablet 3695411d251847b4b1ab0a1eb936a55a
0926 01:55:50.560585 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.560730 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1182768}
I20260926 01:55:50.571485 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 113) took 1195 ms. Trace:
I20260926 01:55:50.554495 9594 consensus_queue.cc:237] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:50.571926 8536 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:50.571990 9514 rpcz_store.cc:276] 0926 01:55:49.376278 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.376396 (+ 118us) service_pool.cc:224] Handling call
0926 01:55:50.571094 (+1194698us) ts_tablet_manager.cc:1151] Deleting tablet 1cba678b66744dcfb4d13ae07ebec33f
0926 01:55:50.571142 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.571428 (+ 286us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1193340}
I20260926 01:55:50.563047 9340 tablet_bootstrap.cc:492] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:50.577540 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 112) took 1201 ms. Trace:
I20260926 01:55:50.577600 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 28eaf7381f364a2682dd07f2bd90ede3 (DEFAULT_TABLE table=default.Table_85df772c6e434f49b9b7b7c88949d303 [id=08a05d07365148b0b9e246f3fcc6d409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.577981 9339 tablet_bootstrap.cc:492] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:50.578380 9339 ts_tablet_manager.cc:1403] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.318s user 0.012s sys 0.029s
I20260926 01:55:50.578626 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28eaf7381f364a2682dd07f2bd90ede3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.580627 9514 rpcz_store.cc:276] 0926 01:55:49.376001 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.397905 (+ 21904us) service_pool.cc:224] Handling call
0926 01:55:50.577164 (+1179259us) ts_tablet_manager.cc:1151] Deleting tablet c36ea09cf15647c0a344c9ba9b289d61
0926 01:55:50.577210 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.577482 (+ 272us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1170555}
I20260926 01:55:50.580797 9339 raft_consensus.cc:359] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.581434 9339 raft_consensus.cc:385] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.581678 9339 raft_consensus.cc:740] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.572907 8536 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:50.586088 9514 tablet_replica.cc:333] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:50.586916 9514 raft_consensus.cc:2243] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:50.587385 9514 raft_consensus.cc:2272] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:50.589217 9514 ts_tablet_manager.cc:1916] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:50.595028 9340 tablet_bootstrap.cc:654] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.597244 9193 tablet_service.cc:1511] Processing CreateTablet for tablet 229f4cdd841949e4a12e8555cdfe42cc (DEFAULT_TABLE table=default.Table_6f6a2a6f414644f7b5d7a322965922fa [id=e7f06f7838a740a09da1cfda5d27fc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.598555 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229f4cdd841949e4a12e8555cdfe42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.598984 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 28eaf7381f364a2682dd07f2bd90ede3 (DEFAULT_TABLE table=default.Table_85df772c6e434f49b9b7b7c88949d303 [id=08a05d07365148b0b9e246f3fcc6d409]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:50.599076 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 59167.2us
I20260926 01:55:50.599988 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28eaf7381f364a2682dd07f2bd90ede3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:50.601037 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 8a54eb3b8cf64a478df38df9d4aef02f because the tablet was not found. No further retry: Not found: Tablet not found: 8a54eb3b8cf64a478df38df9d4aef02f
W20260926 01:55:50.602167 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 9d10be686d9045efaf384d326b356e0e because the tablet was not found. No further retry: Not found: Tablet not found: 9d10be686d9045efaf384d326b356e0e
W20260926 01:55:50.602931 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 3695411d251847b4b1ab0a1eb936a55a because the tablet was not found. No further retry: Not found: Tablet not found: 3695411d251847b4b1ab0a1eb936a55a
W20260926 01:55:50.604832 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 1cba678b66744dcfb4d13ae07ebec33f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cba678b66744dcfb4d13ae07ebec33f already in progress: creating tablet
W20260926 01:55:50.605540 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet c36ea09cf15647c0a344c9ba9b289d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c36ea09cf15647c0a344c9ba9b289d61 already in progress: creating tablet
I20260926 01:55:50.613898 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 229f4cdd841949e4a12e8555cdfe42cc (DEFAULT_TABLE table=default.Table_6f6a2a6f414644f7b5d7a322965922fa [id=e7f06f7838a740a09da1cfda5d27fc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.613984 9339 consensus_queue.cc:260] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.615392 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229f4cdd841949e4a12e8555cdfe42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.630326 9339 ts_tablet_manager.cc:1434] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20260926 01:55:50.630915 9341 tablet_bootstrap.cc:654] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.639029 9046 tablet_service.cc:1511] Processing CreateTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a (DEFAULT_TABLE table=default.Table_37930b35bf904b5aabec0d568896d39b [id=b6747abf32974ff4a3bf5da14e923b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.640172 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f100eb5dc9a8452e943d4cce5284bc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.641310 9195 tablet_service.cc:1511] Processing CreateTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a (DEFAULT_TABLE table=default.Table_37930b35bf904b5aabec0d568896d39b [id=b6747abf32974ff4a3bf5da14e923b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.642489 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f100eb5dc9a8452e943d4cce5284bc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.644076 9339 tablet_bootstrap.cc:492] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:50.648123 9052 tablet_service.cc:1511] Processing CreateTablet for tablet ff004cf835c34095a836ad683cec9256 (DEFAULT_TABLE table=default.Table_0818779c2b1449eaaee8c359e22c3b86 [id=5db31752ef1c4ff992bed8d186ba574e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.649121 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff004cf835c34095a836ad683cec9256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.654834 9683 raft_consensus.cc:493] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.655287 9683 raft_consensus.cc:515] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.657164 9683 leader_election.cc:290] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:50.658965 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b6daaf274894989b90da364f5da5250" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:50.659575 8906 raft_consensus.cc:2468] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:55:50.660614 9006 leader_election.cc:343] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 3b6daaf274894989b90da364f5da5250
I20260926 01:55:50.660966 9198 tablet_service.cc:1511] Processing CreateTablet for tablet ff004cf835c34095a836ad683cec9256 (DEFAULT_TABLE table=default.Table_0818779c2b1449eaaee8c359e22c3b86 [id=5db31752ef1c4ff992bed8d186ba574e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.661257 9003 leader_election.cc:304] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:55:50.662001 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff004cf835c34095a836ad683cec9256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.662160 9683 raft_consensus.cc:2804] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:50.662513 9683 raft_consensus.cc:493] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:50.662809 9683 raft_consensus.cc:3060] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:50.673708 8524 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 68174.6us
I20260926 01:55:50.680447 9341 tablet_bootstrap.cc:492] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:50.680928 9341 ts_tablet_manager.cc:1403] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.152s user 0.010s sys 0.000s
I20260926 01:55:50.684401 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 23d2805b41d645d3bb905f26a380f383
I20260926 01:55:50.684392 9051 tablet_service.cc:1511] Processing CreateTablet for tablet d0bb5090e4874ae188b6d1ffe8649b4e (DEFAULT_TABLE table=default.Table_88a0ad1233b9421a8cdfc42b5b156240 [id=5e6c5a34dd5249689909daf3e142a9b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.685521 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0bb5090e4874ae188b6d1ffe8649b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.685890 8537 catalog_manager.cc:6934] ReplaceTablet: tablet 23d2805b41d645d3bb905f26a380f383 deleted and replaced by tablet 2e12827ad93c47cf9013e238215ceb7a
I20260926 01:55:50.686215 8537 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 176) took 1003 ms. Trace:
I20260926 01:55:50.687973 9341 raft_consensus.cc:359] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.688766 9341 raft_consensus.cc:385] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.689033 9341 raft_consensus.cc:740] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.687511 8537 rpcz_store.cc:276] 0926 01:55:49.682827 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.766795 (+ 83968us) service_pool.cc:224] Handling call
0926 01:55:49.784497 (+ 17702us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60a0e1 00007f717c1d2d37 00007f717c1d2cc2 00007f717c1d2bf1 00007f717c1d2b6c 00007f717c1d1e22 00007f717a45c09c 00007f717a45b4d6
0926 01:55:50.686158 (+901661us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:50.544420 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:50.546029 (+ 1609us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:50.546058 (+ 29us) write_op.cc:435] Acquired schema lock
0926 01:55:50.546073 (+ 15us) tablet.cc:662] Decoding operations
0926 01:55:50.546477 (+ 404us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:50.546524 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:50.546551 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:50.546814 (+ 263us) tablet.cc:701] Row locks acquired
0926 01:55:50.546826 (+ 12us) write_op.cc:260] PREPARE: finished
0926 01:55:50.546933 (+ 107us) write_op.cc:270] Start()
0926 01:55:50.547022 (+ 89us) write_op.cc:276] Timestamp: P: 1790387750546499 usec, L: 0
0926 01:55:50.547037 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:50.547563 (+ 526us) log.cc:844] Serialized 625 byte log entry
0926 01:55:50.676369 (+128806us) op_driver.cc:464] REPLICATION: finished
0926 01:55:50.679659 (+ 3290us) write_op.cc:301] APPLY: starting
0926 01:55:50.679711 (+ 52us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:50.679823 (+ 112us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:50.679911 (+ 88us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:50.680229 (+ 318us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:50.680412 (+ 183us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:50.680452 (+ 40us) write_op.cc:312] APPLY: finished
0926 01:55:50.680634 (+ 182us) log.cc:844] Serialized 29 byte log entry
0926 01:55:50.680758 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:50.680815 (+ 57us) write_op.cc:454] Released schema lock
0926 01:55:50.680842 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":37056896,"child_traces":[["op",{"apply.queue_time_us":3156,"apply.run_cpu_time_us":1427,"apply.run_wall_time_us":7103,"num_ops":2,"prepare.queue_time_us":206559,"prepare.run_cpu_time_us":2296,"prepare.run_wall_time_us":76712,"replication_time_us":129256,"threads_started":1}]]}
I20260926 01:55:50.689690 9341 consensus_queue.cc:260] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.691282 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 23d2805b41d645d3bb905f26a380f383 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e12827ad93c47cf9013e238215ceb7a) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:50.692447 9340 tablet_bootstrap.cc:492] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:50.693075 9340 ts_tablet_manager.cc:1403] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.130s user 0.009s sys 0.002s
I20260926 01:55:50.693675 9047 tablet_service.cc:1558] Processing DeleteTablet for tablet 23d2805b41d645d3bb905f26a380f383 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e12827ad93c47cf9013e238215ceb7a) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:50.695763 9340 raft_consensus.cc:359] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.696389 9340 raft_consensus.cc:385] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.696616 9340 raft_consensus.cc:740] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.702821 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 28eaf7381f364a2682dd07f2bd90ede3 (DEFAULT_TABLE table=default.Table_85df772c6e434f49b9b7b7c88949d303 [id=08a05d07365148b0b9e246f3fcc6d409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.703276 9340 consensus_queue.cc:260] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.703925 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28eaf7381f364a2682dd07f2bd90ede3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.704099 8879 tablet_service.cc:1511] Processing CreateTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a (DEFAULT_TABLE table=default.Table_37930b35bf904b5aabec0d568896d39b [id=b6747abf32974ff4a3bf5da14e923b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.705108 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f100eb5dc9a8452e943d4cce5284bc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.716140 9341 ts_tablet_manager.cc:1434] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260926 01:55:50.725183 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet 23d2805b41d645d3bb905f26a380f383 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e12827ad93c47cf9013e238215ceb7a) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.725889 8881 tablet_service.cc:1511] Processing CreateTablet for tablet ff004cf835c34095a836ad683cec9256 (DEFAULT_TABLE table=default.Table_0818779c2b1449eaaee8c359e22c3b86 [id=5db31752ef1c4ff992bed8d186ba574e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.727108 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff004cf835c34095a836ad683cec9256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.732136 9683 raft_consensus.cc:515] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.735617 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b6daaf274894989b90da364f5da5250" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:50.736239 8906 raft_consensus.cc:3060] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:50.740810 9339 tablet_bootstrap.cc:654] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.744513 8906 raft_consensus.cc:2468] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:50.746029 9003 leader_election.cc:304] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:50.747097 9594 raft_consensus.cc:2804] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:50.747540 9594 raft_consensus.cc:697] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:50.748311 9594 consensus_queue.cc:237] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.754249 9683 leader_election.cc:290] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:50.756470 9006 leader_election.cc:343] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 3b6daaf274894989b90da364f5da5250
I20260926 01:55:50.757339 9340 ts_tablet_manager.cc:1434] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.064s user 0.005s sys 0.000s
I20260926 01:55:50.762588 9523 raft_consensus.cc:493] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:50.763058 9523 raft_consensus.cc:515] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.764997 9523 leader_election.cc:290] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.766796 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1df1d4daaddc4ac4aa02aa7763ea3184" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:50.767441 9072 raft_consensus.cc:2468] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:50.768980 9155 leader_election.cc:304] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:50.769173 8880 tablet_service.cc:1511] Processing CreateTablet for tablet d0bb5090e4874ae188b6d1ffe8649b4e (DEFAULT_TABLE table=default.Table_88a0ad1233b9421a8cdfc42b5b156240 [id=5e6c5a34dd5249689909daf3e142a9b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.770826 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 681900e0fac74cdabaea1d8a7fb4a189 (DEFAULT_TABLE table=default.Table_c30c84e91ae143e3b6918b9be1ec643e [id=d9d802b9f0f34b9399be344e5897a1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.771093 9523 raft_consensus.cc:2804] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:50.771431 9523 raft_consensus.cc:493] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:50.771962 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 681900e0fac74cdabaea1d8a7fb4a189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.771741 9523 raft_consensus.cc:3060] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:50.772008 8886 tablet_service.cc:1511] Processing CreateTablet for tablet 681900e0fac74cdabaea1d8a7fb4a189 (DEFAULT_TABLE table=default.Table_c30c84e91ae143e3b6918b9be1ec643e [id=d9d802b9f0f34b9399be344e5897a1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.772516 9710 tablet_replica.cc:333] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:50.776417 8540 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_ae4f6e78de7a4f6c8f96a3b179cf7c6d" } modify_external_catalogs: true
I20260926 01:55:50.776969 8540 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_ae4f6e78de7a4f6c8f96a3b179cf7c6d" } modify_external_catalogs: true
I20260926 01:55:50.779285 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_88a0ad1233b9421a8cdfc42b5b156240" } new_table_name: "default.Table_aeacfd2450264948a9933ded9cfcabd1" modify_external_catalogs: true
I20260926 01:55:50.779448 9193 tablet_service.cc:1511] Processing CreateTablet for tablet 681900e0fac74cdabaea1d8a7fb4a189 (DEFAULT_TABLE table=default.Table_c30c84e91ae143e3b6918b9be1ec643e [id=d9d802b9f0f34b9399be344e5897a1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.774261 9710 raft_consensus.cc:2243] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:50.779997 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1df1d4daaddc4ac4aa02aa7763ea3184" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:50.780436 9710 raft_consensus.cc:2272] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:50.780619 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 681900e0fac74cdabaea1d8a7fb4a189. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:50.781452 9152 leader_election.cc:343] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:50.782435 9710 ts_tablet_manager.cc:1916] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:50.797186 9340 tablet_bootstrap.cc:492] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:55:50.801662 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 23d2805b41d645d3bb905f26a380f383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23d2805b41d645d3bb905f26a380f383 already in progress: creating tablet
I20260926 01:55:50.761255 9341 tablet_bootstrap.cc:492] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:50.815001 9514 ts_tablet_manager.cc:1929] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:50.819864 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 39cd9130640d4b14beef42e9f1b36b99
I20260926 01:55:50.821534 8532 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 175) took 1330 ms. Trace:
I20260926 01:55:50.822575 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:50.822791 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:50.824455 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 39cd9130640d4b14beef42e9f1b36b99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cd9130640d4b14beef42e9f1b36b99 already in progress: creating tablet
I20260926 01:55:50.821844 8532 rpcz_store.cc:276] 0926 01:55:49.490938 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.644247 (+153309us) service_pool.cc:224] Handling call
0926 01:55:49.710009 (+ 65762us) spinlock_profiling.cc:243] Waited 64.9 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607876 00007f717c1b6b34 00007f717c1b6ac2 00007f717c1b69f1 00007f717c1b696c 00007f717c1b5c22 00007f717a45c09c 00007f717a45b4d6
0926 01:55:49.711279 (+ 1270us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:49.711825 (+ 546us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:49.712000 (+ 175us) catalog_manager.cc:2863] Locking tablets
0926 01:55:50.338431 (+626431us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:50.819421 (+480990us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:50.819650 (+ 229us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:50.819757 (+ 107us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:50.821472 (+ 1715us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:50.621328 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:50.621803 (+ 475us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:50.621834 (+ 31us) write_op.cc:435] Acquired schema lock
0926 01:55:50.621849 (+ 15us) tablet.cc:662] Decoding operations
0926 01:55:50.623134 (+ 1285us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:50.623178 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:50.623201 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:50.623598 (+ 397us) tablet.cc:701] Row locks acquired
0926 01:55:50.623625 (+ 27us) write_op.cc:260] PREPARE: finished
0926 01:55:50.623744 (+ 119us) write_op.cc:270] Start()
0926 01:55:50.623850 (+ 106us) write_op.cc:276] Timestamp: P: 1790387750623301 usec, L: 0
0926 01:55:50.623864 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:50.624402 (+ 538us) log.cc:844] Serialized 759 byte log entry
0926 01:55:50.816297 (+191895us) op_driver.cc:464] REPLICATION: finished
0926 01:55:50.816879 (+ 582us) write_op.cc:301] APPLY: starting
0926 01:55:50.816937 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:50.817051 (+ 114us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:50.817180 (+ 129us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:50.818201 (+ 1021us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:50.818344 (+ 143us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:50.818384 (+ 40us) write_op.cc:312] APPLY: finished
0926 01:55:50.818600 (+ 216us) log.cc:844] Serialized 29 byte log entry
0926 01:55:50.818750 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:50.818814 (+ 64us) write_op.cc:454] Released schema lock
0926 01:55:50.818846 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:50.819080 (+ 234us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:50.819106 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":149341952,"child_traces":[["op",{"apply.queue_time_us":297,"apply.run_cpu_time_us":2183,"apply.run_wall_time_us":2192,"num_ops":2,"prepare.queue_time_us":280794,"prepare.run_cpu_time_us":2600,"prepare.run_wall_time_us":77561,"raft.queue_time_us":2076,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":269,"replication_time_us":192042}]]}
I20260926 01:55:50.826162 9523 raft_consensus.cc:515] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.828490 8535 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 01052a5e22a74a19919bae9dd0296291
I20260926 01:55:50.830750 9514 log.cc:1199] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:50.831564 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 01052a5e22a74a19919bae9dd0296291 with delete_type TABLET_DATA_DELETED (replaced by tablet e8e0e6ede1ce4487b08bfd5743e3656a) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:50.831539 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1df1d4daaddc4ac4aa02aa7763ea3184" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:50.832105 9072 raft_consensus.cc:3060] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:50.835284 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.835644 9514 ts_tablet_manager.cc:1950] T dccbfbc9fb5947f9b85ca8560e205b08 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:50.838580 9710 ts_tablet_manager.cc:1929] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:50.840241 9072 raft_consensus.cc:2468] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:50.836695 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 01052a5e22a74a19919bae9dd0296291 with delete_type TABLET_DATA_DELETED (replaced by tablet e8e0e6ede1ce4487b08bfd5743e3656a) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.841427 9155 leader_election.cc:304] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:50.843327 9635 raft_consensus.cc:2804] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:50.844472 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 117) took 1364 ms. Trace:
I20260926 01:55:50.849308 8535 catalog_manager.cc:6934] ReplaceTablet: tablet 01052a5e22a74a19919bae9dd0296291 deleted and replaced by tablet e8e0e6ede1ce4487b08bfd5743e3656a
I20260926 01:55:50.849704 8535 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 171) took 1590 ms. Trace:
I20260926 01:55:50.852018 9197 tablet_service.cc:1511] Processing CreateTablet for tablet d0bb5090e4874ae188b6d1ffe8649b4e (DEFAULT_TABLE table=default.Table_88a0ad1233b9421a8cdfc42b5b156240 [id=5e6c5a34dd5249689909daf3e142a9b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.849889 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0bb5090e4874ae188b6d1ffe8649b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.849997 8535 rpcz_store.cc:276] 0926 01:55:49.259518 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.259668 (+ 150us) service_pool.cc:224] Handling call
0926 01:55:50.849643 (+1589975us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:50.699008 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:50.699471 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:50.699498 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:55:50.699513 (+ 15us) tablet.cc:662] Decoding operations
0926 01:55:50.699862 (+ 349us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:50.699906 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:50.699932 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:50.700150 (+ 218us) tablet.cc:701] Row locks acquired
0926 01:55:50.700164 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:50.700300 (+ 136us) write_op.cc:270] Start()
0926 01:55:50.700398 (+ 98us) write_op.cc:276] Timestamp: P: 1790387750699847 usec, L: 0
0926 01:55:50.700414 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:50.790528 (+ 90114us) log.cc:844] Serialized 625 byte log entry
0926 01:55:50.793508 (+ 2980us) spinlock_profiling.cc:243] Waited 87.8 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c7895e8 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11 00007f717cc1cd3d 00000000005bd4e4 00000000005bd319
0926 01:55:50.822294 (+ 28786us) op_driver.cc:464] REPLICATION: finished
0926 01:55:50.823644 (+ 1350us) write_op.cc:301] APPLY: starting
0926 01:55:50.823700 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:50.823838 (+ 138us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:50.823859 (+ 21us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:50.824198 (+ 339us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:50.824392 (+ 194us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:50.824433 (+ 41us) write_op.cc:312] APPLY: finished
0926 01:55:50.824604 (+ 171us) log.cc:844] Serialized 29 byte log entry
0926 01:55:50.824739 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:50.824805 (+ 66us) write_op.cc:454] Released schema lock
0926 01:55:50.824839 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:50.828877 (+ 4038us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:50.828895 (+ 18us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24913,"child_traces":[["op",{"apply.queue_time_us":1160,"apply.run_cpu_time_us":1400,"apply.run_wall_time_us":5215,"num_ops":2,"prepare.queue_time_us":302970,"prepare.run_cpu_time_us":4124,"prepare.run_wall_time_us":113395,"raft.queue_time_us":256,"raft.run_cpu_time_us":801,"raft.run_wall_time_us":801,"replication_time_us":121728,"spinlock_wait_cycles":201860992}]]}
I20260926 01:55:50.853086 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0bb5090e4874ae188b6d1ffe8649b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.852609 9514 rpcz_store.cc:276] 0926 01:55:49.480494 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.481099 (+ 605us) service_pool.cc:224] Handling call
0926 01:55:50.585925 (+1104826us) ts_tablet_manager.cc:1151] Deleting tablet dccbfbc9fb5947f9b85ca8560e205b08
0926 01:55:50.585970 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:50.720242 (+134272us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:50.814816 (+ 94574us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:50.844402 (+ 29586us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104008}
I20260926 01:55:50.853238 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 681900e0fac74cdabaea1d8a7fb4a189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.855067 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_011a9ce871e74c2fa862dd3b85e33c32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:50.856818 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_011a9ce871e74c2fa862dd3b85e33c32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:50.860838 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_a21bbc98baa64f09a86dde0520e49cb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:50.862648 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a21bbc98baa64f09a86dde0520e49cb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:50.863533 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet dccbfbc9fb5947f9b85ca8560e205b08 (table default.Table_14869892bcf7429385632e2df181743c [id=bfa8586ef17949bead9731402cd78042]) successfully deleted
I20260926 01:55:50.874248 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 272bb87fe22c4720bcb3f0e07482dd8b (DEFAULT_TABLE table=default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.875391 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272bb87fe22c4720bcb3f0e07482dd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.878024 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 272bb87fe22c4720bcb3f0e07482dd8b (DEFAULT_TABLE table=default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.879108 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272bb87fe22c4720bcb3f0e07482dd8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:50.879448 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 23d2805b41d645d3bb905f26a380f383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23d2805b41d645d3bb905f26a380f383 already in progress: creating tablet
W20260926 01:55:50.881848 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 39cd9130640d4b14beef42e9f1b36b99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cd9130640d4b14beef42e9f1b36b99 already in progress: creating tablet
I20260926 01:55:50.883210 9710 log.cc:1199] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/23d2805b41d645d3bb905f26a380f383
I20260926 01:55:50.890444 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 01052a5e22a74a19919bae9dd0296291 with delete_type TABLET_DATA_DELETED (replaced by tablet e8e0e6ede1ce4487b08bfd5743e3656a) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:50.891309 9711 tablet_replica.cc:333] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:50.892319 9711 raft_consensus.cc:2243] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:50.892808 9711 raft_consensus.cc:2272] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:50.894279 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1df1d4daaddc4ac4aa02aa7763ea3184" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:50.897686 9711 ts_tablet_manager.cc:1916] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:50.898983 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 272bb87fe22c4720bcb3f0e07482dd8b (DEFAULT_TABLE table=default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.900050 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 272bb87fe22c4720bcb3f0e07482dd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.901608 8531 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:50.904654 9340 tablet_bootstrap.cc:654] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.914963 9514 tablet_replica.cc:333] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:50.916074 9514 raft_consensus.cc:2243] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:50.916939 9514 raft_consensus.cc:2272] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:50.918866 9514 ts_tablet_manager.cc:1916] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:50.902415 8531 catalog_manager.cc:5838] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:50.921559 8531 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:50.937497 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 943a990fa10948e8961c02e9999569f5 (DEFAULT_TABLE table=default.Table_3907f5fafff14aec94d0a9e331e86dc9 [id=cc59eec581574b65a6332863423dab90]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:50.937729 8536 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 36) took 1605 ms (client timeout 1000 ms). Trace:
I20260926 01:55:50.938764 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943a990fa10948e8961c02e9999569f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.883466 9523 leader_election.cc:290] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:50.914635 9196 tablet_service.cc:1511] Processing CreateTablet for tablet f1f020590a8b428788e2bb516a11cc45 (DEFAULT_TABLE table=default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.941895 9052 tablet_service.cc:1511] Processing CreateTablet for tablet f1f020590a8b428788e2bb516a11cc45 (DEFAULT_TABLE table=default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.942438 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1f020590a8b428788e2bb516a11cc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.924587 8886 tablet_service.cc:1511] Processing CreateTablet for tablet f1f020590a8b428788e2bb516a11cc45 (DEFAULT_TABLE table=default.Table_8969d22a4ab54a5484838555a31bb684 [id=2c5fb89a77a6445e8aef61c7eaf0603f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:50.925701 9152 leader_election.cc:343] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:50.947576 9152 consensus_peers.cc:597] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:50.884696 9710 ts_tablet_manager.cc:1950] T 23d2805b41d645d3bb905f26a380f383 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:50.948232 9339 tablet_bootstrap.cc:492] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:50.948661 9339 ts_tablet_manager.cc:1403] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.317s user 0.005s sys 0.007s
I20260926 01:55:50.951880 9339 raft_consensus.cc:359] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.952513 9339 raft_consensus.cc:385] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:50.952765 9339 raft_consensus.cc:740] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:50.953385 9339 consensus_queue.cc:260] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:50.926118 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f with delete_type TABLET_DATA_DELETED (replaced by tablet 17b8b0d54f5645e5963f1de217f94c84) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.937214 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.954370 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.938408 8878 tablet_service.cc:1558] Processing DeleteTablet for tablet a85e47c1bf9f4e96958aa2de41039180 with delete_type TABLET_DATA_DELETED (replaced by tablet f1f020590a8b428788e2bb516a11cc45) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.940094 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:50.955128 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1f020590a8b428788e2bb516a11cc45. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:50.938062 8536 rpcz_store.cc:269] 0926 01:55:49.332610 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.340759 (+ 8149us) service_pool.cc:224] Handling call
0926 01:55:50.937642 (+1596883us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:50.812694 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:50.813450 (+ 756us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:50.813502 (+ 52us) write_op.cc:435] Acquired schema lock
0926 01:55:50.813519 (+ 17us) tablet.cc:662] Decoding operations
0926 01:55:50.814429 (+ 910us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:50.814585 (+ 156us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:50.814719 (+ 134us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:50.815953 (+ 1234us) tablet.cc:701] Row locks acquired
0926 01:55:50.815969 (+ 16us) write_op.cc:260] PREPARE: finished
0926 01:55:50.816373 (+ 404us) write_op.cc:270] Start()
0926 01:55:50.816673 (+ 300us) write_op.cc:276] Timestamp: P: 1790387750815821 usec, L: 0
0926 01:55:50.816793 (+ 120us) op_driver.cc:348] REPLICATION: starting
0926 01:55:50.820516 (+ 3723us) log.cc:844] Serialized 907 byte log entry
0926 01:55:50.868855 (+ 48339us) spinlock_profiling.cc:243] Waited 871 us on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c7895e8 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11 00007f717cc1cd3d 00000000005bd4e4 00000000005bd319
0926 01:55:50.887136 (+ 18281us) op_driver.cc:464] REPLICATION: finished
0926 01:55:50.887379 (+ 243us) write_op.cc:301] APPLY: starting
0926 01:55:50.887441 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:50.887564 (+ 123us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:50.887579 (+ 15us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:50.887826 (+ 247us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:50.887956 (+ 130us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:50.888004 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:55:50.888192 (+ 188us) log.cc:844] Serialized 29 byte log entry
0926 01:55:50.888913 (+ 721us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:50.888983 (+ 70us) write_op.cc:454] Released schema lock
0926 01:55:50.889024 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":27178,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":1297,"apply.run_wall_time_us":1839,"num_ops":2,"prepare.queue_time_us":216312,"prepare.run_cpu_time_us":4109,"prepare.run_wall_time_us":56448,"raft.queue_time_us":95,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":323,"replication_time_us":70217,"spinlock_wait_cycles":2004224}]]}
I20260926 01:55:50.960035 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_ba018a2d02f94e7bb7c54c3496475dac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:50.961692 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba018a2d02f94e7bb7c54c3496475dac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:50.885793 9341 tablet_bootstrap.cc:654] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:50.967526 9339 ts_tablet_manager.cc:1434] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260926 01:55:50.967869 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 23d2805b41d645d3bb905f26a380f383 (table default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]) successfully deleted
W20260926 01:55:50.969722 9155 consensus_peers.cc:597] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:50.974962 9635 raft_consensus.cc:697] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:50.975502 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 943a990fa10948e8961c02e9999569f5 (DEFAULT_TABLE table=default.Table_3907f5fafff14aec94d0a9e331e86dc9 [id=cc59eec581574b65a6332863423dab90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.976241 9635 consensus_queue.cc:237] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:50.977741 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 39cd9130640d4b14beef42e9f1b36b99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cd9130640d4b14beef42e9f1b36b99 already in progress: creating tablet
I20260926 01:55:50.984527 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 943a990fa10948e8961c02e9999569f5 (DEFAULT_TABLE table=default.Table_3907f5fafff14aec94d0a9e331e86dc9 [id=cc59eec581574b65a6332863423dab90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:50.985667 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943a990fa10948e8961c02e9999569f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:50.988071 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0bb5090e4874ae188b6d1ffe8649b4e consensus state has no leader
I20260926 01:55:50.991982 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1f020590a8b428788e2bb516a11cc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:50.993363 9339 tablet_bootstrap.cc:492] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:50.978785 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943a990fa10948e8961c02e9999569f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.004522 9341 tablet_bootstrap.cc:492] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:50.998549 9710 tablet_replica.cc:333] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:51.004979 9341 ts_tablet_manager.cc:1403] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.244s user 0.009s sys 0.002s
I20260926 01:55:51.005604 9710 raft_consensus.cc:2243] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:51.006933 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_5bb843a35ff742499a87f56bb5d77d32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:51.007617 9341 raft_consensus.cc:359] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.007839 8878 tablet_service.cc:1511] Processing CreateTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 (DEFAULT_TABLE table=default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.008266 9341 raft_consensus.cc:385] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.008512 9341 raft_consensus.cc:740] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:55:51.008611 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bb843a35ff742499a87f56bb5d77d32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:51.009006 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1743e863cdd84f3f8ca413fa81cd1914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.009526 9710 pending_rounds.cc:70] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Trying to abort 1 pending ops.
I20260926 01:55:51.014562 9710 pending_rounds.cc:77] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428223648247808 op_type: NO_OP noop_request { }
I20260926 01:55:51.015094 9710 raft_consensus.cc:2889] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:51.015533 9710 raft_consensus.cc:2272] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:51.017705 9710 ts_tablet_manager.cc:1916] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:51.002586 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 (DEFAULT_TABLE table=default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.025931 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1743e863cdd84f3f8ca413fa81cd1914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.027484 9340 tablet_bootstrap.cc:492] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:51.027992 9340 ts_tablet_manager.cc:1403] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.231s user 0.005s sys 0.008s
I20260926 01:55:51.031337 9340 raft_consensus.cc:359] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.031980 9340 raft_consensus.cc:385] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.032233 9340 raft_consensus.cc:740] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.034829 9340 consensus_queue.cc:260] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.040562 8540 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:55:51.042897 8540 master_service.cc:960] ReplaceTablet: received request to replace tablet 1df1d4daaddc4ac4aa02aa7763ea3184 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:51.044731 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:51.052709 9341 consensus_queue.cc:260] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.055351 9341 ts_tablet_manager.cc:1434] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20260926 01:55:51.056217 9341 tablet_bootstrap.cc:492] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:51.077108 9193 tablet_service.cc:1511] Processing CreateTablet for tablet c9f54f648a55404fb6622b35d84368ee (DEFAULT_TABLE table=default.Table_6ba9495c267245aa86a1b9184b17a7f1 [id=162b3018a1a24d3391f815bd2efc8e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.078642 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9f54f648a55404fb6622b35d84368ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.079542 9594 raft_consensus.cc:493] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.080792 9594 raft_consensus.cc:515] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.084126 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 (DEFAULT_TABLE table=default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.085292 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1743e863cdd84f3f8ca413fa81cd1914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.058504 9683 raft_consensus.cc:493] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.085760 9683 raft_consensus.cc:515] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.091367 8906 raft_consensus.cc:3060] T 6d101ad03f8e4d7f8bf70ea993c4f58b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.103928 8881 tablet_service.cc:1511] Processing CreateTablet for tablet c9f54f648a55404fb6622b35d84368ee (DEFAULT_TABLE table=default.Table_6ba9495c267245aa86a1b9184b17a7f1 [id=162b3018a1a24d3391f815bd2efc8e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.105427 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9f54f648a55404fb6622b35d84368ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.108114 9703 consensus_queue.cc:1048] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:51.120849 9340 ts_tablet_manager.cc:1434] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.093s user 0.005s sys 0.000s
I20260926 01:55:51.122041 9340 tablet_bootstrap.cc:492] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:51.127172 9711 ts_tablet_manager.cc:1929] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:51.127703 9711 log.cc:1199] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/01052a5e22a74a19919bae9dd0296291
I20260926 01:55:51.127982 9594 leader_election.cc:290] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.128945 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53c9eb8a6d648b2a9359f02b7f8764b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:51.130085 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:51.130519 9003 leader_election.cc:343] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:51.130753 8904 raft_consensus.cc:2468] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:51.131810 9003 leader_election.cc:304] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:51.132944 9683 leader_election.cc:290] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.133435 9683 raft_consensus.cc:2804] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:51.133741 9683 raft_consensus.cc:493] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:51.134193 9683 raft_consensus.cc:3060] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.134495 8535 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_3907f5fafff14aec94d0a9e331e86dc9" } modify_external_catalogs: true
I20260926 01:55:51.135093 8535 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_3907f5fafff14aec94d0a9e331e86dc9" } modify_external_catalogs: true
I20260926 01:55:51.141415 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:51.154896 9711 ts_tablet_manager.cc:1950] T 01052a5e22a74a19919bae9dd0296291 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:51.161489 8878 tablet_service.cc:1511] Processing CreateTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.163048 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.126371 9514 ts_tablet_manager.cc:1929] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:51.164780 9514 log.cc:1199] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/01052a5e22a74a19919bae9dd0296291
I20260926 01:55:51.165565 9514 ts_tablet_manager.cc:1950] T 01052a5e22a74a19919bae9dd0296291 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:51.178834 9341 tablet_bootstrap.cc:654] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.178932 9635 raft_consensus.cc:493] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.180240 9635 raft_consensus.cc:515] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.195410 9341 tablet_bootstrap.cc:492] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:51.195919 9341 ts_tablet_manager.cc:1403] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.001s
I20260926 01:55:51.197566 9635 leader_election.cc:290] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:51.199303 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:51.201692 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17b8b0d54f5645e5963f1de217f94c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.203027 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17b8b0d54f5645e5963f1de217f94c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.203191 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53c9eb8a6d648b2a9359f02b7f8764b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:51.203663 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:51.203882 9218 raft_consensus.cc:2468] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:51.207429 9341 raft_consensus.cc:359] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.208042 9341 raft_consensus.cc:385] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.208278 9341 raft_consensus.cc:740] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.212668 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 01052a5e22a74a19919bae9dd0296291 (table default.Table_a8c0c28267ee48f39ee4eb4c11da8a9c [id=6a3cae01c8664f3d8d4b89e8a91665f7]) successfully deleted
I20260926 01:55:51.212416 8534 catalog_manager.cc:5838] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:51.214183 8534 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:51.215128 8534 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:51.208882 9341 consensus_queue.cc:260] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.216284 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:51.216892 9218 raft_consensus.cc:2468] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:51.221939 8906 raft_consensus.cc:2468] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:51.223479 9152 leader_election.cc:304] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:51.226608 9051 tablet_service.cc:1511] Processing CreateTablet for tablet c9f54f648a55404fb6622b35d84368ee (DEFAULT_TABLE table=default.Table_6ba9495c267245aa86a1b9184b17a7f1 [id=162b3018a1a24d3391f815bd2efc8e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.227547 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 56b0e40e5e964b90bd2dd158bbc4497a (DEFAULT_TABLE table=default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.228065 9635 raft_consensus.cc:2804] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:51.228389 9635 raft_consensus.cc:493] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:51.228673 9635 raft_consensus.cc:3060] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.228809 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b0e40e5e964b90bd2dd158bbc4497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.228667 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 01052a5e22a74a19919bae9dd0296291 (table default.Table_a8c0c28267ee48f39ee4eb4c11da8a9c [id=6a3cae01c8664f3d8d4b89e8a91665f7]) successfully deleted
W20260926 01:55:51.229372 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf2fa5fbe4146b5adb27ff09de878d3 already in progress: creating tablet
I20260926 01:55:51.229525 9594 consensus_queue.cc:1048] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:51.231277 9006 leader_election.cc:304] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:51.236613 9339 tablet_bootstrap.cc:654] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.227710 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.238718 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 56b0e40e5e964b90bd2dd158bbc4497a (DEFAULT_TABLE table=default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.240427 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b0e40e5e964b90bd2dd158bbc4497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.240769 9710 ts_tablet_manager.cc:1929] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:51.241545 9683 raft_consensus.cc:515] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.243397 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:51.247087 9340 tablet_bootstrap.cc:654] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.247730 9594 raft_consensus.cc:2804] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:51.247916 9072 raft_consensus.cc:2393] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:51.248108 9594 raft_consensus.cc:493] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:51.248411 9594 raft_consensus.cc:3060] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.242996 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17b8b0d54f5645e5963f1de217f94c84. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:51.250077 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 8a011a9edfe94e04afb6ea0367dd979f because the tablet was not found. No further retry: Not found: Tablet not found: 8a011a9edfe94e04afb6ea0367dd979f
W20260926 01:55:51.252847 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet dccbfbc9fb5947f9b85ca8560e205b08 because the tablet was not found. No further retry: Not found: Tablet not found: dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:51.254492 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9f54f648a55404fb6622b35d84368ee. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:51.254495 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 32e0fe60b8744dbb8ae2bafefe3e834d because the tablet was not found. No further retry: Not found: Tablet not found: 32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:51.255906 9683 leader_election.cc:290] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:51.256958 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet a85e47c1bf9f4e96958aa2de41039180 because the tablet was not found. No further retry: Not found: Tablet not found: a85e47c1bf9f4e96958aa2de41039180
I20260926 01:55:51.256716 9594 raft_consensus.cc:515] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.257859 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:51.258558 8906 raft_consensus.cc:3060] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.259466 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53c9eb8a6d648b2a9359f02b7f8764b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:51.260845 9003 leader_election.cc:343] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:51.265587 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 56b0e40e5e964b90bd2dd158bbc4497a (DEFAULT_TABLE table=default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.266158 9710 log.cc:1199] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/01052a5e22a74a19919bae9dd0296291
I20260926 01:55:51.267050 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b0e40e5e964b90bd2dd158bbc4497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.243917 9341 ts_tablet_manager.cc:1434] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
I20260926 01:55:51.268543 9341 tablet_bootstrap.cc:492] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:51.282231 9341 tablet_bootstrap.cc:654] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.283360 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 661148692f9e4127af9abf5cc6cc0f54 (DEFAULT_TABLE table=default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.284740 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 661148692f9e4127af9abf5cc6cc0f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.287106 9499 raft_consensus.cc:493] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.287650 9499 raft_consensus.cc:515] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:51.291538 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfaca3fc02634f459083d23f3102f725" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:51.292151 9072 raft_consensus.cc:2468] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:51.295276 9499 leader_election.cc:290] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.295943 8843 leader_election.cc:304] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:55:51.297662 8843 leader_election.cc:343] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: dfaca3fc02634f459083d23f3102f725
I20260926 01:55:51.298684 9499 raft_consensus.cc:2804] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:51.299091 9499 raft_consensus.cc:493] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:51.299507 9499 raft_consensus.cc:3060] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.303333 8532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 183) took 1018 ms. Trace:
I20260926 01:55:51.303715 8532 rpcz_store.cc:276] 0926 01:55:50.284828 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:50.833941 (+549113us) service_pool.cc:224] Handling call
0926 01:55:50.860738 (+ 26797us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607566 00007f717c1b2034 00007f717c1b1fc2 00007f717c1b1ef1 00007f717c1b1e6c 00007f717c1b1122 00007f717a45c09c 00007f717a45b4d6
0926 01:55:50.864454 (+ 3716us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:55:50.866081 (+ 1627us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:55:51.302645 (+436564us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:55:51.302964 (+ 319us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:55:51.303260 (+ 296us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:51.238833 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:51.244705 (+ 5872us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:51.244732 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:55:51.244744 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:51.245025 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:51.245071 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:51.245097 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:51.245299 (+ 202us) tablet.cc:701] Row locks acquired
0926 01:55:51.245314 (+ 15us) write_op.cc:260] PREPARE: finished
0926 01:55:51.245451 (+ 137us) write_op.cc:270] Start()
0926 01:55:51.245549 (+ 98us) write_op.cc:276] Timestamp: P: 1790387751245382 usec, L: 0
0926 01:55:51.245575 (+ 26us) op_driver.cc:348] REPLICATION: starting
0926 01:55:51.246273 (+ 698us) log.cc:844] Serialized 433 byte log entry
0926 01:55:51.298240 (+ 51967us) op_driver.cc:464] REPLICATION: finished
0926 01:55:51.299507 (+ 1267us) write_op.cc:301] APPLY: starting
0926 01:55:51.299592 (+ 85us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:51.299716 (+ 124us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:51.299812 (+ 96us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:51.300741 (+ 929us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:51.300883 (+ 142us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:55:51.300924 (+ 41us) write_op.cc:312] APPLY: finished
0926 01:55:51.301128 (+ 204us) log.cc:844] Serialized 29 byte log entry
0926 01:55:51.301293 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:51.301371 (+ 78us) write_op.cc:454] Released schema lock
0926 01:55:51.301405 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":59823488,"child_traces":[["op",{"apply.queue_time_us":1082,"apply.run_cpu_time_us":2119,"apply.run_wall_time_us":2119,"num_ops":2,"prepare.queue_time_us":287344,"raft.queue_time_us":1358,"raft.run_cpu_time_us":795,"raft.run_wall_time_us":797,"replication_time_us":52557}]]}
I20260926 01:55:51.307765 9499 raft_consensus.cc:515] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:51.311002 9514 tablet_replica.cc:333] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:51.320084 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfaca3fc02634f459083d23f3102f725" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:51.320679 9072 raft_consensus.cc:3060] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.267043 9710 ts_tablet_manager.cc:1950] T 01052a5e22a74a19919bae9dd0296291 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:51.318420 9514 raft_consensus.cc:2243] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:51.329355 9514 raft_consensus.cc:2272] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:51.337636 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 01052a5e22a74a19919bae9dd0296291 (table default.Table_a8c0c28267ee48f39ee4eb4c11da8a9c [id=6a3cae01c8664f3d8d4b89e8a91665f7]) successfully deleted
I20260926 01:55:51.341722 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:51.345875 9499 leader_election.cc:290] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.343847 9072 raft_consensus.cc:2468] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:51.348456 9514 ts_tablet_manager.cc:1916] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:51.356611 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 661148692f9e4127af9abf5cc6cc0f54 (DEFAULT_TABLE table=default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.357898 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 661148692f9e4127af9abf5cc6cc0f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.360069 9195 tablet_service.cc:1511] Processing CreateTablet for tablet c448caeeedfb4fc88eee88e40eebfeeb (DEFAULT_TABLE table=default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.362061 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c448caeeedfb4fc88eee88e40eebfeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.296319 9635 raft_consensus.cc:515] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.363251 9219 raft_consensus.cc:2393] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:51.369890 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:51.370098 9635 leader_election.cc:290] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:51.371130 9072 raft_consensus.cc:2393] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:51.379649 8906 raft_consensus.cc:2468] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
W20260926 01:55:51.382743 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf2fa5fbe4146b5adb27ff09de878d3 already in progress: creating tablet
I20260926 01:55:51.385402 9341 tablet_bootstrap.cc:492] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:51.384346 9339 tablet_bootstrap.cc:492] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:51.385869 9341 ts_tablet_manager.cc:1403] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.118s user 0.013s sys 0.012s
I20260926 01:55:51.386129 9339 ts_tablet_manager.cc:1403] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.393s user 0.008s sys 0.005s
I20260926 01:55:51.389792 9340 tablet_bootstrap.cc:492] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:51.389514 9339 raft_consensus.cc:359] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.390187 9339 raft_consensus.cc:385] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.402675 9340 ts_tablet_manager.cc:1403] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.281s user 0.008s sys 0.003s
I20260926 01:55:51.406211 9340 raft_consensus.cc:359] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.421116 9341 raft_consensus.cc:359] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.421865 9341 raft_consensus.cc:385] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.422130 9341 raft_consensus.cc:740] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.413221 9052 tablet_service.cc:1511] Processing CreateTablet for tablet c448caeeedfb4fc88eee88e40eebfeeb (DEFAULT_TABLE table=default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.430136 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c448caeeedfb4fc88eee88e40eebfeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.431094 9341 consensus_queue.cc:260] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.416934 9594 leader_election.cc:290] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.441964 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53c9eb8a6d648b2a9359f02b7f8764b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:51.443385 9219 raft_consensus.cc:3060] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.390431 9339 raft_consensus.cc:740] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.450616 9219 raft_consensus.cc:2468] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:51.455441 9341 ts_tablet_manager.cc:1434] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.069s user 0.006s sys 0.001s
I20260926 01:55:51.456512 9341 tablet_bootstrap.cc:492] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:51.457463 9339 consensus_queue.cc:260] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.461558 9339 ts_tablet_manager.cc:1434] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.075s user 0.004s sys 0.003s
I20260926 01:55:51.472445 9339 tablet_bootstrap.cc:492] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:51.471398 9006 leader_election.cc:304] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:51.486233 9594 raft_consensus.cc:2804] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:51.480664 9339 tablet_bootstrap.cc:654] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.491289 9594 raft_consensus.cc:697] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:51.492806 9594 consensus_queue.cc:237] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.504292 9341 tablet_bootstrap.cc:654] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.539959 8537 master_service.cc:960] ReplaceTablet: received request to replace tablet 1743e863cdd84f3f8ca413fa81cd1914 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:51.553206 8535 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 943a990fa10948e8961c02e9999569f5
I20260926 01:55:51.555430 8535 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8969d22a4ab54a5484838555a31bb684" } new_table_name: "default.Table_d4b013d8752449f898dd39794a37a01f" modify_external_catalogs: true
I20260926 01:55:51.563400 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 943a990fa10948e8961c02e9999569f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:51 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:51.565356 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 943a990fa10948e8961c02e9999569f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943a990fa10948e8961c02e9999569f5 already in progress: creating tablet
I20260926 01:55:51.565356 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 943a990fa10948e8961c02e9999569f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:51 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:51.567366 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 943a990fa10948e8961c02e9999569f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943a990fa10948e8961c02e9999569f5 already in progress: creating tablet
I20260926 01:55:51.574436 9339 tablet_bootstrap.cc:492] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:51.575001 9339 ts_tablet_manager.cc:1403] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.103s user 0.015s sys 0.004s
I20260926 01:55:51.577783 9339 raft_consensus.cc:359] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.578504 9339 raft_consensus.cc:385] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.578751 9339 raft_consensus.cc:740] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.578891 9703 raft_consensus.cc:493] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.579300 9703 raft_consensus.cc:515] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.579540 9339 consensus_queue.cc:260] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.581372 9703 leader_election.cc:290] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.581503 9339 ts_tablet_manager.cc:1434] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 01:55:51.582605 9339 tablet_bootstrap.cc:492] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:55:51.586110 8524 consensus_peers.cc:597] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba -> Peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Couldn't send request to peer f75ba46482de422ea23daf17aa100849. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260926 01:55:51.586416 9341 tablet_bootstrap.cc:492] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:51.586920 9341 ts_tablet_manager.cc:1403] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.002s
I20260926 01:55:51.589350 9341 raft_consensus.cc:359] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.589967 9341 raft_consensus.cc:385] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.590204 9341 raft_consensus.cc:740] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.591972 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75d4598596ff49378fc3536fff0ac1b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:51.593133 9219 raft_consensus.cc:2468] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:51.594376 9006 leader_election.cc:304] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:51.596210 9703 raft_consensus.cc:2804] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:51.596539 9703 raft_consensus.cc:493] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:51.597035 9703 raft_consensus.cc:3060] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.599120 9339 tablet_bootstrap.cc:654] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:51.603032 8525 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d0bb5090e4874ae188b6d1ffe8649b4e consensus state has no leader
I20260926 01:55:51.603333 9341 consensus_queue.cc:260] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.605706 9341 ts_tablet_manager.cc:1434] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.019s user 0.003s sys 0.002s
I20260926 01:55:51.611076 9703 raft_consensus.cc:515] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.612766 9635 raft_consensus.cc:493] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.613227 9635 raft_consensus.cc:515] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.615113 9635 leader_election.cc:290] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:51.616858 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:51.618523 9155 leader_election.cc:343] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:51.619043 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75d4598596ff49378fc3536fff0ac1b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:51.619570 9219 raft_consensus.cc:3060] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.622552 9703 leader_election.cc:290] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.612886 9341 tablet_bootstrap.cc:492] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:51.627621 9339 tablet_bootstrap.cc:492] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:51.628044 9339 ts_tablet_manager.cc:1403] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.000s
I20260926 01:55:51.630654 9339 raft_consensus.cc:359] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.631263 9339 raft_consensus.cc:385] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.631507 9339 raft_consensus.cc:740] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.632145 9339 consensus_queue.cc:260] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.644646 9339 ts_tablet_manager.cc:1434] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260926 01:55:51.645949 9339 tablet_bootstrap.cc:492] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:51.652035 9339 tablet_bootstrap.cc:654] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.674623 9594 consensus_queue.cc:1048] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:51.678825 9340 raft_consensus.cc:385] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.679312 9340 raft_consensus.cc:740] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.680042 9340 consensus_queue.cc:260] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.706048 9339 tablet_bootstrap.cc:492] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:51.706681 9339 ts_tablet_manager.cc:1403] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.003s
I20260926 01:55:51.710119 9635 raft_consensus.cc:493] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.710515 8540 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 1df1d4daaddc4ac4aa02aa7763ea3184
I20260926 01:55:51.710784 9635 raft_consensus.cc:515] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.711995 8540 catalog_manager.cc:6934] ReplaceTablet: tablet 1df1d4daaddc4ac4aa02aa7763ea3184 deleted and replaced by tablet 0c6cccff3de64f3384ce58bb84c43487
I20260926 01:55:51.712291 9219 raft_consensus.cc:2468] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:51.714203 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:51.714955 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c6cccff3de64f3384ce58bb84c43487) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:51.715664 9635 leader_election.cc:290] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:51.715723 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c6cccff3de64f3384ce58bb84c43487) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:51.716574 9710 tablet_replica.cc:333] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:51.716986 9006 leader_election.cc:304] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:51.717466 9710 raft_consensus.cc:2243] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:51.718247 9710 pending_rounds.cc:70] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Trying to abort 1 pending ops.
I20260926 01:55:51.718492 9710 pending_rounds.cc:77] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428228177973248 op_type: NO_OP noop_request { }
I20260926 01:55:51.718863 9710 raft_consensus.cc:2889] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:51.718932 9703 raft_consensus.cc:2804] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:51.719205 9710 raft_consensus.cc:2272] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:51.719376 9703 raft_consensus.cc:697] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:51.720013 9711 tablet_replica.cc:333] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:51.720158 9703 consensus_queue.cc:237] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.720899 9711 raft_consensus.cc:2243] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:51.721345 9711 raft_consensus.cc:2272] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:51.727152 9711 ts_tablet_manager.cc:1916] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:51.734378 9710 ts_tablet_manager.cc:1916] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:51.748119 9340 ts_tablet_manager.cc:1434] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.345s user 0.010s sys 0.004s
I20260926 01:55:51.753142 8540 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_db78756a42ab4ab3924a6b3d285f430b" } modify_external_catalogs: true
I20260926 01:55:51.753926 8540 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_db78756a42ab4ab3924a6b3d285f430b" } modify_external_catalogs: true
I20260926 01:55:51.750357 9341 tablet_bootstrap.cc:654] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:51.757014 9155 leader_election.cc:343] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:51.769654 9339 raft_consensus.cc:359] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.770434 9339 raft_consensus.cc:385] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.770881 9339 raft_consensus.cc:740] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.771724 9339 consensus_queue.cc:260] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.788035 9339 ts_tablet_manager.cc:1434] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.081s user 0.007s sys 0.000s
I20260926 01:55:51.792371 9339 tablet_bootstrap.cc:492] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:51.801827 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:51.803333 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:51.804263 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:51.806252 9742 raft_consensus.cc:493] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.806892 9742 raft_consensus.cc:515] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.815678 9341 tablet_bootstrap.cc:492] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:51.816267 9341 ts_tablet_manager.cc:1403] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.204s user 0.007s sys 0.005s
I20260926 01:55:51.816399 9339 tablet_bootstrap.cc:654] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.819818 8538 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 2 got 0
I20260926 01:55:51.821785 8538 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:51.827628 9341 raft_consensus.cc:359] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.828351 9341 raft_consensus.cc:385] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.828600 9341 raft_consensus.cc:740] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.829458 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:55:51.830984 8537 catalog_manager.cc:6934] ReplaceTablet: tablet 1743e863cdd84f3f8ca413fa81cd1914 deleted and replaced by tablet b778c95327fc477ca210cb3c983665c1
I20260926 01:55:51.831939 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_db38e199ec7b405f87d487e2ff6d24f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:51.833580 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db38e199ec7b405f87d487e2ff6d24f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:51.835567 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 with delete_type TABLET_DATA_DELETED (replaced by tablet b778c95327fc477ca210cb3c983665c1) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:51.836643 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 with delete_type TABLET_DATA_DELETED (replaced by tablet b778c95327fc477ca210cb3c983665c1) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:51.836756 9341 consensus_queue.cc:260] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.837739 9339 tablet_bootstrap.cc:492] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:51.838173 9339 ts_tablet_manager.cc:1403] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.000s
I20260926 01:55:51.840716 9339 raft_consensus.cc:359] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.841347 9339 raft_consensus.cc:385] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:51.841609 9339 raft_consensus.cc:740] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:51.841866 9635 raft_consensus.cc:493] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.842283 9635 raft_consensus.cc:515] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.844246 9635 leader_election.cc:290] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:51.845666 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55452e70a7fe48efbafdc7cbb29c6272" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:51.850751 8538 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:51.851009 9341 ts_tablet_manager.cc:1434] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20260926 01:55:51.860157 9339 consensus_queue.cc:260] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.861275 9072 raft_consensus.cc:2468] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
W20260926 01:55:51.862105 8535 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1f020590a8b428788e2bb516a11cc45 consensus state has no leader
I20260926 01:55:51.863200 9155 leader_election.cc:304] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:51.863430 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_68c0da73fcc54bf99603db86570ccbe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:51.865008 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68c0da73fcc54bf99603db86570ccbe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:51.865208 9635 raft_consensus.cc:2804] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:51.865635 9635 raft_consensus.cc:493] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:51.866007 9635 raft_consensus.cc:3060] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.868017 9341 tablet_bootstrap.cc:492] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:51.873140 9006 consensus_peers.cc:597] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b6daaf274894989b90da364f5da5250. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:51.880957 9340 tablet_bootstrap.cc:492] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:51.809245 9742 leader_election.cc:290] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:51.884320 9339 ts_tablet_manager.cc:1434] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20260926 01:55:51.891728 9339 tablet_bootstrap.cc:492] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:51.892951 9713 raft_consensus.cc:493] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:51.893417 9713 raft_consensus.cc:515] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.895468 9713 leader_election.cc:290] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:51.896353 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75d4598596ff49378fc3536fff0ac1b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:51.897183 8904 raft_consensus.cc:2468] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:51.899860 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet 943a990fa10948e8961c02e9999569f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:51 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:51.899998 9635 raft_consensus.cc:515] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.901633 8883 tablet_service.cc:1511] Processing CreateTablet for tablet c448caeeedfb4fc88eee88e40eebfeeb (DEFAULT_TABLE table=default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.890568 9003 leader_election.cc:304] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:55:51.902886 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55452e70a7fe48efbafdc7cbb29c6272" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:51.903453 9071 raft_consensus.cc:3060] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.897301 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:51.904085 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 661148692f9e4127af9abf5cc6cc0f54 (DEFAULT_TABLE table=default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:51.904712 9703 raft_consensus.cc:2804] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:51.905160 9703 raft_consensus.cc:697] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:51.905900 9703 consensus_queue.cc:237] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:51.909233 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23d2805b41d645d3bb905f26a380f383" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:51.909839 9069 raft_consensus.cc:2468] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:51.911290 9071 raft_consensus.cc:2468] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:55:51.914149 9155 leader_election.cc:343] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:51.914901 9155 leader_election.cc:304] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:51.917755 9713 raft_consensus.cc:2804] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:51.920955 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3bb1643da054583b97efd2e1a9e9c2f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:51.921732 8907 raft_consensus.cc:2393] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:51.923183 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75d4598596ff49378fc3536fff0ac1b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:51.923739 8906 raft_consensus.cc:3060] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:51.936686 9711 ts_tablet_manager.cc:1929] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:51.936956 9341 tablet_bootstrap.cc:654] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:51.937145 9711 log.cc:1199] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/1df1d4daaddc4ac4aa02aa7763ea3184
I20260926 01:55:51.937794 9711 ts_tablet_manager.cc:1950] T 1df1d4daaddc4ac4aa02aa7763ea3184 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:51.945782 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c448caeeedfb4fc88eee88e40eebfeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.949569 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 661148692f9e4127af9abf5cc6cc0f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:51.952596 9635 leader_election.cc:290] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:51.959559 8843 leader_election.cc:304] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:51.960386 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 with delete_type TABLET_DATA_DELETED (replaced by tablet b778c95327fc477ca210cb3c983665c1) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:51.961642 9740 raft_consensus.cc:2804] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:51.962628 9740 raft_consensus.cc:697] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:51.963511 9740 consensus_queue.cc:237] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:51.968896 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55452e70a7fe48efbafdc7cbb29c6272" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:51.972227 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:51.974367 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:51.982414 8540 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet cc66312cd4454669bf0e35234bd3dac4
I20260926 01:55:51.962843 8843 leader_election.cc:304] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:51.987206 9740 raft_consensus.cc:2804] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260926 01:55:51.987697 8843 leader_election.cc:343] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: dfaca3fc02634f459083d23f3102f725
I20260926 01:55:51.987982 9740 raft_consensus.cc:493] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:51.988306 9740 raft_consensus.cc:3060] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:51.988603 8843 leader_election.cc:343] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 23d2805b41d645d3bb905f26a380f383
W20260926 01:55:51.997895 9003 consensus_peers.cc:597] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32e0fe60b8744dbb8ae2bafefe3e834d. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:52.000952 9725 raft_consensus.cc:493] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.000847 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55452e70a7fe48efbafdc7cbb29c6272" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:52.001480 9725 raft_consensus.cc:515] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.003080 9152 leader_election.cc:304] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.004186 9725 leader_election.cc:290] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.004863 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:52.005456 9218 raft_consensus.cc:2468] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:52.006721 9006 leader_election.cc:304] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:52.007551 9725 raft_consensus.cc:2804] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:52.007897 9725 raft_consensus.cc:493] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:52.008203 9725 raft_consensus.cc:3060] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:52.011349 9152 leader_election.cc:343] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.011312 9594 consensus_queue.cc:1048] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:52.012725 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:52.015630 9635 raft_consensus.cc:2749] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:55:52.016345 9340 tablet_bootstrap.cc:654] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:52.017637 9341 tablet_bootstrap.cc:492] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:52.020015 9341 ts_tablet_manager.cc:1403] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.153s user 0.016s sys 0.008s
I20260926 01:55:52.023721 9341 raft_consensus.cc:359] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.031617 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_d1175fab270f48b19ec70a1957bd6af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:52.036056 9514 ts_tablet_manager.cc:1929] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:52.036317 9339 tablet_bootstrap.cc:654] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:52.036688 9514 log.cc:1199] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/1866cd97bb024a9baecf101a6d5b1ee4
I20260926 01:55:52.043218 9341 raft_consensus.cc:385] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 01:55:52.044054 9152 leader_election.cc:343] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.044449 9152 leader_election.cc:304] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.043572 9341 raft_consensus.cc:740] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.045238 9753 raft_consensus.cc:2749] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:55:52.046005 9152 leader_election.cc:343] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.046392 9152 leader_election.cc:304] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.045873 9341 consensus_queue.cc:260] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.051712 9341 ts_tablet_manager.cc:1434] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20260926 01:55:52.052224 9753 raft_consensus.cc:2749] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:52.052668 9341 tablet_bootstrap.cc:492] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:52.053710 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_a8580d0d4cfe42c594511d26fbcff4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:52.062935 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8580d0d4cfe42c594511d26fbcff4b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:52.063597 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 1df1d4daaddc4ac4aa02aa7763ea3184 (table default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]) successfully deleted
I20260926 01:55:52.042016 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c6cccff3de64f3384ce58bb84c43487) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:52.067433 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 1743e863cdd84f3f8ca413fa81cd1914 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1743e863cdd84f3f8ca413fa81cd1914 already in progress: creating tablet
I20260926 01:55:52.067406 9514 ts_tablet_manager.cc:1950] T 1866cd97bb024a9baecf101a6d5b1ee4 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:52.071143 9341 tablet_bootstrap.cc:654] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:52.029793 9710 ts_tablet_manager.cc:1929] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:52.073210 9710 log.cc:1199] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/1df1d4daaddc4ac4aa02aa7763ea3184
I20260926 01:55:52.074070 9710 ts_tablet_manager.cc:1950] T 1df1d4daaddc4ac4aa02aa7763ea3184 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:52.079104 8906 raft_consensus.cc:2468] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:52.079213 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 1df1d4daaddc4ac4aa02aa7763ea3184 (table default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]) successfully deleted
W20260926 01:55:52.083961 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 1743e863cdd84f3f8ca413fa81cd1914 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1743e863cdd84f3f8ca413fa81cd1914 already in progress: creating tablet
I20260926 01:55:52.114523 9740 raft_consensus.cc:515] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.115422 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:52.116564 9740 leader_election.cc:290] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.116448 9052 tablet_service.cc:1511] Processing CreateTablet for tablet e18ddaa6f0e748cf930e29cbca3844b1 (DEFAULT_TABLE table=default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.117569 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18ddaa6f0e748cf930e29cbca3844b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.119334 9339 tablet_bootstrap.cc:492] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:52.119771 9339 ts_tablet_manager.cc:1403] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.228s user 0.011s sys 0.001s
I20260926 01:55:52.123795 9198 tablet_service.cc:1511] Processing CreateTablet for tablet e18ddaa6f0e748cf930e29cbca3844b1 (DEFAULT_TABLE table=default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.124847 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18ddaa6f0e748cf930e29cbca3844b1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:52.125479 9152 leader_election.cc:343] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:52.126852 9152 leader_election.cc:343] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.127205 9152 leader_election.cc:304] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.127873 9635 raft_consensus.cc:2749] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:52.130880 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 135) took 1128 ms. Trace:
I20260926 01:55:52.131294 9514 rpcz_store.cc:276] 0926 01:55:50.939825 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:50.939953 (+ 128us) service_pool.cc:224] Handling call
0926 01:55:51.310801 (+370848us) ts_tablet_manager.cc:1151] Deleting tablet 1866cd97bb024a9baecf101a6d5b1ee4
0926 01:55:51.310860 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:51.785327 (+474467us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:51.934343 (+149016us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:52.068378 (+134035us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":355485}
I20260926 01:55:52.132916 9514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 138) took 1089 ms. Trace:
I20260926 01:55:52.133208 9514 rpcz_store.cc:276] 0926 01:55:51.043351 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:51.243087 (+199736us) service_pool.cc:224] Handling call
0926 01:55:52.132444 (+889357us) ts_tablet_manager.cc:1151] Deleting tablet 3bf2fa5fbe4146b5adb27ff09de878d3
0926 01:55:52.132533 (+ 89us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:52.132848 (+ 315us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":888581}
I20260926 01:55:52.133395 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
W20260926 01:55:52.136251 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1175fab270f48b19ec70a1957bd6af6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:52.136720 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:52.136898 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 4623456312284a698ae69d4054909647 (DEFAULT_TABLE table=default.Table_632a5e0f37424fc9a71a1b75d1a4814c [id=2103f22c921d4213827d63854da752aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.138118 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4623456312284a698ae69d4054909647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.146179 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 4623456312284a698ae69d4054909647 (DEFAULT_TABLE table=default.Table_632a5e0f37424fc9a71a1b75d1a4814c [id=2103f22c921d4213827d63854da752aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.147440 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4623456312284a698ae69d4054909647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.148319 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23d2805b41d645d3bb905f26a380f383" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:52.148906 9073 raft_consensus.cc:3060] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:52.150661 9003 leader_election.cc:343] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.122360 9339 raft_consensus.cc:359] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.152289 9339 raft_consensus.cc:385] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:52.152590 9339 raft_consensus.cc:740] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.153306 9339 consensus_queue.cc:260] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.154582 9713 raft_consensus.cc:697] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:52.154773 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:52.155622 9713 consensus_queue.cc:237] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.156782 8878 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58226 (request call id 140) took 1001 ms. Trace:
I20260926 01:55:52.156805 9073 raft_consensus.cc:2468] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:52.157155 8878 rpcz_store.cc:276] 0926 01:55:51.155821 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:51.160668 (+ 4847us) service_pool.cc:224] Handling call
0926 01:55:51.198160 (+ 37492us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:51.201269 (+ 3109us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:51.807140 (+605871us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:51.933889 (+126749us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:52.156599 (+222710us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:52.160837 9725 raft_consensus.cc:515] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.162982 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
W20260926 01:55:52.164422 8539 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8e0e6ede1ce4487b08bfd5743e3656a consensus state has no leader
I20260926 01:55:52.164984 8539 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 168) took 3032 ms. Trace:
I20260926 01:55:52.166380 9339 ts_tablet_manager.cc:1434] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260926 01:55:52.167353 9339 tablet_bootstrap.cc:492] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:52.165298 8539 rpcz_store.cc:276] 0926 01:55:49.132609 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.132747 (+ 138us) service_pool.cc:224] Handling call
0926 01:55:49.222642 (+ 89895us) spinlock_profiling.cc:243] Waited 88.9 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607be6 00007f717c1bb634 00007f717c1bb5c2 00007f717c1bb4f1 00007f717c1bb46c 00007f717c1ba722 00007f717a45c09c 00007f717a45b4d6
0926 01:55:49.223456 (+ 814us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:49.224161 (+ 705us) catalog_manager.cc:3764] Apply alter schema
0926 01:55:49.224784 (+ 623us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:55:49.224967 (+ 183us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:55:49.535737 (+310770us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:55:52.164905 (+2629168us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:49.438120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:49.439353 (+ 1233us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:49.439379 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:49.439391 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:49.439704 (+ 313us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:49.439752 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:49.439784 (+ 32us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:49.439950 (+ 166us) tablet.cc:701] Row locks acquired
0926 01:55:49.439964 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:49.440095 (+ 131us) write_op.cc:270] Start()
0926 01:55:49.440184 (+ 89us) write_op.cc:276] Timestamp: P: 1790387749440041 usec, L: 0
0926 01:55:49.440198 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:49.440738 (+ 540us) log.cc:844] Serialized 405 byte log entry
0926 01:55:49.526952 (+ 86214us) op_driver.cc:464] REPLICATION: finished
0926 01:55:49.533924 (+ 6972us) write_op.cc:301] APPLY: starting
0926 01:55:49.533989 (+ 65us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:49.534082 (+ 93us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:49.534166 (+ 84us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:49.534399 (+ 233us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:49.534541 (+ 142us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:49.534584 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:55:49.534771 (+ 187us) log.cc:844] Serialized 23 byte log entry
0926 01:55:49.535447 (+ 676us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:49.535511 (+ 64us) write_op.cc:454] Released schema lock
0926 01:55:49.535542 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:49.536057 (+ 515us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:49.536080 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":204715008,"child_traces":[["op",{"apply.queue_time_us":6787,"apply.run_cpu_time_us":1269,"apply.run_wall_time_us":2107,"num_ops":1,"prepare.queue_time_us":211905,"prepare.run_cpu_time_us":2280,"prepare.run_wall_time_us":80553,"raft.queue_time_us":73142,"raft.run_cpu_time_us":903,"raft.run_wall_time_us":904,"replication_time_us":86646}]]}
I20260926 01:55:52.171479 8539 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_37930b35bf904b5aabec0d568896d39b" } modify_external_catalogs: true
I20260926 01:55:52.172108 8539 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_37930b35bf904b5aabec0d568896d39b" } modify_external_catalogs: true
I20260926 01:55:52.173775 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:52.185084 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 92896db04df346bf91f732b1ce89cd23 (DEFAULT_TABLE table=default.Table_129e4100f7f245e9877924fd11902aea [id=6aedab17761246a7aace66b2cfd6d09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.186144 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92896db04df346bf91f732b1ce89cd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.189074 9340 tablet_bootstrap.cc:492] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:52.189568 9340 ts_tablet_manager.cc:1403] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.309s user 0.007s sys 0.007s
I20260926 01:55:52.192261 9340 raft_consensus.cc:359] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:52.193488 9003 leader_election.cc:343] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.199070 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 1866cd97bb024a9baecf101a6d5b1ee4 (table default.Table_8e5e15ce8e58425697a7f013c09c0c72 [id=5dd626e9121c4d5aba51637d7c6cb8a2]) successfully deleted
W20260926 01:55:52.201864 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf2fa5fbe4146b5adb27ff09de878d3 already in progress: creating tablet
I20260926 01:55:52.217674 9703 raft_consensus.cc:493] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.218211 9703 raft_consensus.cc:515] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.219733 9341 tablet_bootstrap.cc:492] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:52.220207 9341 ts_tablet_manager.cc:1403] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.168s user 0.017s sys 0.000s
I20260926 01:55:52.220446 8540 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:52.222637 9751 raft_consensus.cc:493] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.223129 9751 raft_consensus.cc:515] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.235220 9635 raft_consensus.cc:493] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.235627 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:52.235751 9635 raft_consensus.cc:515] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.236917 9751 leader_election.cc:290] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:52.237680 9155 leader_election.cc:343] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.238211 9635 leader_election.cc:290] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:52.239614 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:52.240576 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 92896db04df346bf91f732b1ce89cd23 (DEFAULT_TABLE table=default.Table_129e4100f7f245e9877924fd11902aea [id=6aedab17761246a7aace66b2cfd6d09e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:52.241143 9155 leader_election.cc:343] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.241772 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92896db04df346bf91f732b1ce89cd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.254326 8540 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:52.255139 8531 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_a4d2c1ed01c74ff3a7260703edd509be [id=2af856781c904c77a11ca9f4966ca44b] alter complete (version 3)
I20260926 01:55:52.273694 8539 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:55:52.275825 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:52 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:52.342878 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 943a990fa10948e8961c02e9999569f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 943a990fa10948e8961c02e9999569f5 already in progress: creating tablet
W20260926 01:55:52.343837 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 1743e863cdd84f3f8ca413fa81cd1914 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1743e863cdd84f3f8ca413fa81cd1914 already in progress: creating tablet
W20260926 01:55:52.344858 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df1d4daaddc4ac4aa02aa7763ea3184 already in progress: creating tablet
I20260926 01:55:52.249963 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:52.258347 8878 tablet_service.cc:1511] Processing CreateTablet for tablet e18ddaa6f0e748cf930e29cbca3844b1 (DEFAULT_TABLE table=default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.357251 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18ddaa6f0e748cf930e29cbca3844b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.264129 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:52.266782 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 4623456312284a698ae69d4054909647 (DEFAULT_TABLE table=default.Table_632a5e0f37424fc9a71a1b75d1a4814c [id=2103f22c921d4213827d63854da752aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.363314 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4623456312284a698ae69d4054909647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.282572 8886 tablet_service.cc:1511] Processing CreateTablet for tablet 92896db04df346bf91f732b1ce89cd23 (DEFAULT_TABLE table=default.Table_129e4100f7f245e9877924fd11902aea [id=6aedab17761246a7aace66b2cfd6d09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.367245 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92896db04df346bf91f732b1ce89cd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.376492 9339 tablet_bootstrap.cc:654] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:52.386965 9152 leader_election.cc:343] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.387388 9152 leader_election.cc:304] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.388108 9713 raft_consensus.cc:2749] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:55:52.388806 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet f100eb5dc9a8452e943d4cce5284bc8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f100eb5dc9a8452e943d4cce5284bc8a already in progress: creating tablet
I20260926 01:55:52.389602 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 0fdb7a7328c54031bd3afac1636d0f75 (DEFAULT_TABLE table=default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.390699 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdb7a7328c54031bd3afac1636d0f75. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:52.390867 9152 leader_election.cc:343] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.391220 9152 leader_election.cc:304] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.391858 9713 raft_consensus.cc:2749] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:52.393535 9725 leader_election.cc:290] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.396293 9195 tablet_service.cc:1558] Processing DeleteTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:52 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:52.396775 9340 raft_consensus.cc:385] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:52.397059 9340 raft_consensus.cc:740] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.397133 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_269f661dac8f41d08eebbdc0cd122fee" } new_table_name: "default.Table_ee6cc37e9698474a8a3194f060b8c028" modify_external_catalogs: true
I20260926 01:55:52.397661 9340 consensus_queue.cc:260] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:52.398082 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet f100eb5dc9a8452e943d4cce5284bc8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f100eb5dc9a8452e943d4cce5284bc8a already in progress: creating tablet
W20260926 01:55:52.345803 8843 leader_election.cc:343] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 23d2805b41d645d3bb905f26a380f383
W20260926 01:55:52.406673 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 60918.8us
I20260926 01:55:52.408388 9725 consensus_queue.cc:1048] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:52.408851 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:52.409369 9218 raft_consensus.cc:3060] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:52.414921 9341 raft_consensus.cc:359] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.416015 9725 raft_consensus.cc:493] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.416522 9725 raft_consensus.cc:515] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.431818 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:52.432859 9725 leader_election.cc:290] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.427229 9703 leader_election.cc:290] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:52.433410 9003 leader_election.cc:343] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.436143 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:52.415656 9341 raft_consensus.cc:385] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 01:55:52.438777 9003 leader_election.cc:343] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.438818 9341 raft_consensus.cc:740] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.428797 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:52.439549 9341 consensus_queue.cc:260] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.439975 9220 raft_consensus.cc:2468] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:52.435783 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:52.416970 9218 raft_consensus.cc:2468] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:52.441368 9217 raft_consensus.cc:2468] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:52.441607 9341 ts_tablet_manager.cc:1434] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.221s user 0.005s sys 0.000s
I20260926 01:55:52.442591 9006 leader_election.cc:304] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:52.443723 9006 leader_election.cc:304] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:52.444793 9006 leader_election.cc:304] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:52.445683 9683 raft_consensus.cc:2804] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:52.448642 9340 ts_tablet_manager.cc:1434] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.259s user 0.005s sys 0.000s
I20260926 01:55:52.450258 9340 tablet_bootstrap.cc:492] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:52.456419 9703 raft_consensus.cc:2804] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260926 01:55:52.456796 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 49511us
I20260926 01:55:52.456930 9703 raft_consensus.cc:493] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:52.457427 9703 raft_consensus.cc:3060] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:52.457468 8843 leader_election.cc:304] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:55:52.458412 9740 raft_consensus.cc:2804] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:52.459510 9740 raft_consensus.cc:697] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:55:52.460300 9740 consensus_queue.cc:237] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.465950 9767 raft_consensus.cc:2804] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:52.466390 9767 raft_consensus.cc:493] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:52.466864 9767 raft_consensus.cc:3060] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:52.486181 9683 raft_consensus.cc:697] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:52.487197 9341 tablet_bootstrap.cc:492] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:52.487802 9683 consensus_queue.cc:237] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.488011 9767 raft_consensus.cc:515] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.494575 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:52 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:52.495239 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 0fdb7a7328c54031bd3afac1636d0f75 (DEFAULT_TABLE table=default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:52.496536 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet f100eb5dc9a8452e943d4cce5284bc8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f100eb5dc9a8452e943d4cce5284bc8a already in progress: creating tablet
W20260926 01:55:52.497084 9003 consensus_peers.cc:597] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:52.497342 9767 leader_election.cc:290] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.497498 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdb7a7328c54031bd3afac1636d0f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.498879 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:52.499548 9219 raft_consensus.cc:3060] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:52.500495 9339 tablet_bootstrap.cc:492] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:52.500906 9339 ts_tablet_manager.cc:1403] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.334s user 0.008s sys 0.006s
I20260926 01:55:52.512455 9339 raft_consensus.cc:359] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.514133 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:52.517025 9003 leader_election.cc:343] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.521720 9219 raft_consensus.cc:2468] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:52.523082 9768 consensus_queue.cc:1048] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:55:52.523566 9703 raft_consensus.cc:515] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.530496 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:52.532039 9003 leader_election.cc:343] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.514321 9339 raft_consensus.cc:385] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:52.533735 9006 leader_election.cc:304] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:52.533969 9339 raft_consensus.cc:740] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.534911 9683 raft_consensus.cc:2804] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:52.534715 9339 consensus_queue.cc:260] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.536791 9713 raft_consensus.cc:493] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.537305 9713 raft_consensus.cc:515] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.537662 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 0fdb7a7328c54031bd3afac1636d0f75 (DEFAULT_TABLE table=default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.539022 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdb7a7328c54031bd3afac1636d0f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.539521 9713 leader_election.cc:290] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:52.541108 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:55:52.544415 9155 leader_election.cc:343] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.544735 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 16f49a5a10ea4f7a821d89f18e3af560 (DEFAULT_TABLE table=default.Table_f820adee11bf4557b6119642992da842 [id=d16775efe3d1459a9e1d3152f2b945f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.545866 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f49a5a10ea4f7a821d89f18e3af560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.546224 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 6d56f3a5dfed4ac78f1da92e22db558a (DEFAULT_TABLE table=default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.547510 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d56f3a5dfed4ac78f1da92e22db558a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.547536 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 16f49a5a10ea4f7a821d89f18e3af560 (DEFAULT_TABLE table=default.Table_f820adee11bf4557b6119642992da842 [id=d16775efe3d1459a9e1d3152f2b945f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.550765 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f49a5a10ea4f7a821d89f18e3af560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.535513 9683 raft_consensus.cc:697] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:52.555462 9683 consensus_queue.cc:237] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.563900 9703 leader_election.cc:290] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.567101 9339 ts_tablet_manager.cc:1434] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20260926 01:55:52.568722 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:52.569890 9341 tablet_bootstrap.cc:654] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:52.572934 9152 leader_election.cc:343] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.573360 9152 leader_election.cc:304] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.574606 9635 raft_consensus.cc:2749] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:52.585909 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 72f666b1c069435b8ff0a3fae9f1e873 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:52.586170 8537 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:52.586741 8537 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:52.588052 8537 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:52.586169 9767 consensus_queue.cc:1048] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:52.591969 9339 tablet_bootstrap.cc:492] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:52.604966 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:52.605720 9218 raft_consensus.cc:3060] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:52.616777 9339 tablet_bootstrap.cc:654] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:52.616722 8879 tablet_service.cc:1511] Processing CreateTablet for tablet 16f49a5a10ea4f7a821d89f18e3af560 (DEFAULT_TABLE table=default.Table_f820adee11bf4557b6119642992da842 [id=d16775efe3d1459a9e1d3152f2b945f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.618000 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f49a5a10ea4f7a821d89f18e3af560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.619249 9740 raft_consensus.cc:493] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.619696 9740 raft_consensus.cc:515] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.621647 9740 leader_election.cc:290] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.623224 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 6d56f3a5dfed4ac78f1da92e22db558a (DEFAULT_TABLE table=default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.624276 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d56f3a5dfed4ac78f1da92e22db558a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:52.627596 8843 leader_election.cc:343] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 1df1d4daaddc4ac4aa02aa7763ea3184
W20260926 01:55:52.628450 8843 leader_election.cc:343] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 1df1d4daaddc4ac4aa02aa7763ea3184
I20260926 01:55:52.628819 8843 leader_election.cc:304] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:52.630632 9740 raft_consensus.cc:2749] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:52.641403 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 6d56f3a5dfed4ac78f1da92e22db558a (DEFAULT_TABLE table=default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.642650 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d56f3a5dfed4ac78f1da92e22db558a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.644973 8537 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:52.655481 9339 tablet_bootstrap.cc:492] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:52.655951 9339 ts_tablet_manager.cc:1403] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.001s
I20260926 01:55:52.658589 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 1f10cb68586b4a73abb86641b90c12f7 (DEFAULT_TABLE table=default.Table_84d6c2cf62bb42779b018cfc68f2b42f [id=8e41fb91eafb42aa9cf99435ebb8371f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.659662 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f10cb68586b4a73abb86641b90c12f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.662245 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 1f10cb68586b4a73abb86641b90c12f7 (DEFAULT_TABLE table=default.Table_84d6c2cf62bb42779b018cfc68f2b42f [id=8e41fb91eafb42aa9cf99435ebb8371f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.663290 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f10cb68586b4a73abb86641b90c12f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.664529 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 1c8761a6c3f7435193a91bfc4fe7fa86 (DEFAULT_TABLE table=default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:52.612432 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_39a7b6f090bf44b5b0c15825bf0d75e5 [id=6a3cae01c8664f3d8d4b89e8a91665f7], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8e0e6ede1ce4487b08bfd5743e3656a consensus state has no leader
I20260926 01:55:52.667656 9341 tablet_bootstrap.cc:492] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:52.668102 9341 ts_tablet_manager.cc:1403] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.181s user 0.007s sys 0.004s
I20260926 01:55:52.667860 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 1f10cb68586b4a73abb86641b90c12f7 (DEFAULT_TABLE table=default.Table_84d6c2cf62bb42779b018cfc68f2b42f [id=8e41fb91eafb42aa9cf99435ebb8371f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.669051 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f10cb68586b4a73abb86641b90c12f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.672264 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 1c8761a6c3f7435193a91bfc4fe7fa86 (DEFAULT_TABLE table=default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.673314 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8761a6c3f7435193a91bfc4fe7fa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.675892 9683 raft_consensus.cc:493] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.676396 9683 raft_consensus.cc:515] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.679888 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:52.681823 9003 leader_election.cc:343] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.663334 8537 catalog_manager.cc:5838] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:52.678359 9683 leader_election.cc:290] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.690078 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:52.690883 9217 raft_consensus.cc:2468] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:52.692984 9006 leader_election.cc:304] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:52.694199 9767 raft_consensus.cc:2804] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:52.688390 8535 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 72f666b1c069435b8ff0a3fae9f1e873
I20260926 01:55:52.694591 9767 raft_consensus.cc:493] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:52.694911 9767 raft_consensus.cc:3060] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:52.696890 8535 catalog_manager.cc:6934] ReplaceTablet: tablet 72f666b1c069435b8ff0a3fae9f1e873 deleted and replaced by tablet e76071c66b1d4635ac1a28110eb684fa
I20260926 01:55:52.696856 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:52.699088 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 72f666b1c069435b8ff0a3fae9f1e873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72f666b1c069435b8ff0a3fae9f1e873 already in progress: creating tablet
I20260926 01:55:52.699903 8887 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:52.701896 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 72f666b1c069435b8ff0a3fae9f1e873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72f666b1c069435b8ff0a3fae9f1e873 already in progress: creating tablet
I20260926 01:55:52.703392 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:52.718686 9341 raft_consensus.cc:359] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.719424 9341 raft_consensus.cc:385] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:52.719687 9341 raft_consensus.cc:740] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.720327 9341 consensus_queue.cc:260] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.749403 9341 ts_tablet_manager.cc:1434] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.081s user 0.007s sys 0.000s
I20260926 01:55:52.760237 9767 raft_consensus.cc:515] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.763747 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:52.765614 9003 leader_election.cc:343] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.768720 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8761a6c3f7435193a91bfc4fe7fa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.774353 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 880d77d42c394e6a9cf0b3768a915402 (DEFAULT_TABLE table=default.Table_235a08a4f73a4761935c03af88c3a393 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.706272 9218 raft_consensus.cc:2468] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:52.776439 9006 leader_election.cc:304] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:52.776373 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 880d77d42c394e6a9cf0b3768a915402 (DEFAULT_TABLE table=default.Table_235a08a4f73a4761935c03af88c3a393 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.777565 9594 raft_consensus.cc:2804] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:52.777697 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 880d77d42c394e6a9cf0b3768a915402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.702350 9340 tablet_bootstrap.cc:654] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:52.779685 9594 raft_consensus.cc:697] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:52.780859 9594 consensus_queue.cc:237] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.781896 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 1c8761a6c3f7435193a91bfc4fe7fa86 (DEFAULT_TABLE table=default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:52.782828 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 72f666b1c069435b8ff0a3fae9f1e873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72f666b1c069435b8ff0a3fae9f1e873 already in progress: creating tablet
I20260926 01:55:52.782999 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8761a6c3f7435193a91bfc4fe7fa86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.782100 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 880d77d42c394e6a9cf0b3768a915402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.790131 8879 tablet_service.cc:1511] Processing CreateTablet for tablet e8e0e6ede1ce4487b08bfd5743e3656a (DEFAULT_TABLE table=default.Table_39a7b6f090bf44b5b0c15825bf0d75e5 [id=6a3cae01c8664f3d8d4b89e8a91665f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.791416 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e0e6ede1ce4487b08bfd5743e3656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.796258 9193 tablet_service.cc:1511] Processing CreateTablet for tablet e8e0e6ede1ce4487b08bfd5743e3656a (DEFAULT_TABLE table=default.Table_39a7b6f090bf44b5b0c15825bf0d75e5 [id=6a3cae01c8664f3d8d4b89e8a91665f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.797423 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e0e6ede1ce4487b08bfd5743e3656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.801362 9051 tablet_service.cc:1511] Processing CreateTablet for tablet e8e0e6ede1ce4487b08bfd5743e3656a (DEFAULT_TABLE table=default.Table_39a7b6f090bf44b5b0c15825bf0d75e5 [id=6a3cae01c8664f3d8d4b89e8a91665f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.807451 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e0e6ede1ce4487b08bfd5743e3656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.812143 9340 tablet_bootstrap.cc:492] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:52.812680 9340 ts_tablet_manager.cc:1403] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.363s user 0.011s sys 0.001s
I20260926 01:55:52.817238 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 880d77d42c394e6a9cf0b3768a915402 (DEFAULT_TABLE table=default.Table_235a08a4f73a4761935c03af88c3a393 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.817690 9340 raft_consensus.cc:359] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.768383 9339 raft_consensus.cc:359] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.822834 9339 raft_consensus.cc:385] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:52.823326 9339 raft_consensus.cc:740] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.822543 9340 raft_consensus.cc:385] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:52.823977 9340 raft_consensus.cc:740] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:52.824183 9339 consensus_queue.cc:260] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.825744 9340 consensus_queue.cc:260] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.828752 9767 leader_election.cc:290] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.828951 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 880d77d42c394e6a9cf0b3768a915402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.828979 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 2e12827ad93c47cf9013e238215ceb7a (DEFAULT_TABLE table=default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.829999 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e12827ad93c47cf9013e238215ceb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.845834 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 2e12827ad93c47cf9013e238215ceb7a (DEFAULT_TABLE table=default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.819300 9341 tablet_bootstrap.cc:492] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:52.835983 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:52.846721 9217 raft_consensus.cc:3060] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:52.847045 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e12827ad93c47cf9013e238215ceb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.853972 9217 raft_consensus.cc:2468] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:52.855571 9006 leader_election.cc:304] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:52.856678 9767 raft_consensus.cc:2804] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:52.857172 9767 raft_consensus.cc:697] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:52.864871 9341 tablet_bootstrap.cc:654] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:52.875175 8535 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:52.876466 8535 catalog_manager.cc:5838] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:52.877852 8535 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:52.878692 8535 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:55:52.868363 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 0d068e93fd374a6091d7473db069a7fa (DEFAULT_TABLE table=default.Table_fb988b16dc1843a7b22bced528ea5ea6 [id=99516dec436f449cb3df511cd26cecad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.886217 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d068e93fd374a6091d7473db069a7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.869498 9683 consensus_queue.cc:1048] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260926 01:55:52.858590 9767 consensus_queue.cc:237] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.895576 9683 raft_consensus.cc:493] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.896051 9683 raft_consensus.cc:515] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.898059 9683 leader_election.cc:290] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.913516 9340 ts_tablet_manager.cc:1434] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.100s user 0.003s sys 0.003s
W20260926 01:55:52.925333 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 85845.7us
I20260926 01:55:52.884826 9739 raft_consensus.cc:493] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:52.878618 9053 tablet_service.cc:1511] Processing CreateTablet for tablet dbf222f9e14d4332b679f7ee94af1e0e (DEFAULT_TABLE table=default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:52.931197 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf222f9e14d4332b679f7ee94af1e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.930063 9739 raft_consensus.cc:515] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.944279 9739 leader_election.cc:290] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:52.870178 9339 ts_tablet_manager.cc:1434] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.214s user 0.006s sys 0.001s
I20260926 01:55:52.945567 9339 tablet_bootstrap.cc:492] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:52.946707 9340 tablet_bootstrap.cc:492] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:52.961134 9794 tablet.cc:1751] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:52.967478 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:52.971551 9003 leader_election.cc:343] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:52.980742 9340 tablet_bootstrap.cc:654] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:52.997658 9594 consensus_queue.cc:1048] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:55:52.975068 9339 tablet_bootstrap.cc:654] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.017915 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 2e12827ad93c47cf9013e238215ceb7a (DEFAULT_TABLE table=default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.019192 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e12827ad93c47cf9013e238215ceb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:53.023280 8883 tablet_service.cc:1511] Processing CreateTablet for tablet dbf222f9e14d4332b679f7ee94af1e0e (DEFAULT_TABLE table=default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.024211 9339 tablet_bootstrap.cc:492] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.024290 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf222f9e14d4332b679f7ee94af1e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:53.024626 9339 ts_tablet_manager.cc:1403] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.000s
I20260926 01:55:53.017915 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 0d068e93fd374a6091d7473db069a7fa (DEFAULT_TABLE table=default.Table_fb988b16dc1843a7b22bced528ea5ea6 [id=99516dec436f449cb3df511cd26cecad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.027570 9339 raft_consensus.cc:359] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.028196 9339 raft_consensus.cc:385] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.028446 9339 raft_consensus.cc:740] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.028569 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d068e93fd374a6091d7473db069a7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:53.030746 9339 consensus_queue.cc:260] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:52.874576 9194 tablet_service.cc:1511] Processing CreateTablet for tablet 0d068e93fd374a6091d7473db069a7fa (DEFAULT_TABLE table=default.Table_fb988b16dc1843a7b22bced528ea5ea6 [id=99516dec436f449cb3df511cd26cecad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.032855 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d068e93fd374a6091d7473db069a7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:52.909469 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:53.035861 9339 ts_tablet_manager.cc:1434] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260926 01:55:52.969622 9198 tablet_service.cc:1511] Processing CreateTablet for tablet dbf222f9e14d4332b679f7ee94af1e0e (DEFAULT_TABLE table=default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.037416 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbf222f9e14d4332b679f7ee94af1e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:53.041025 8535 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:53.042946 9339 tablet_bootstrap.cc:492] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:55:53.043627 9006 leader_election.cc:343] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:53.043452 9798 tablet.cc:1751] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:53.044025 9006 leader_election.cc:304] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:53.045128 9594 raft_consensus.cc:2749] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:53.055263 9339 tablet_bootstrap.cc:654] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.077642 9339 tablet_bootstrap.cc:492] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.078102 9339 ts_tablet_manager.cc:1403] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.003s
I20260926 01:55:53.080972 9339 raft_consensus.cc:359] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.081640 9339 raft_consensus.cc:385] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.081895 9339 raft_consensus.cc:740] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.082533 9339 consensus_queue.cc:260] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.087159 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:53.092024 9339 ts_tablet_manager.cc:1434] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20260926 01:55:53.092903 9339 tablet_bootstrap.cc:492] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:53.095297 9594 raft_consensus.cc:493] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:53.095739 9594 raft_consensus.cc:515] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.097040 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09351c3db384b4383ca5ff0eaca982c" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:53.097779 9219 raft_consensus.cc:2393] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:53.097682 9594 leader_election.cc:290] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:53.099291 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:53.099825 9219 raft_consensus.cc:2468] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:53.101020 9006 leader_election.cc:304] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:53.102068 9703 raft_consensus.cc:2804] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:53.102415 9703 raft_consensus.cc:493] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:53.102789 9703 raft_consensus.cc:3060] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.104417 8534 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204] alter complete (version 3)
I20260926 01:55:53.107823 8539 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 197) took 1041 ms. Trace:
I20260926 01:55:53.107959 9341 tablet_bootstrap.cc:492] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:53.108387 9341 ts_tablet_manager.cc:1403] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.358s user 0.007s sys 0.002s
I20260926 01:55:53.108148 8539 rpcz_store.cc:276] 0926 01:55:52.066504 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:52.396773 (+330269us) service_pool.cc:224] Handling call
0926 01:55:52.397693 (+ 920us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:52.471929 (+ 74236us) catalog_manager.cc:3764] Apply alter schema
0926 01:55:52.472578 (+ 649us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:55:52.472821 (+ 243us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:55:52.627061 (+154240us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:55:53.107752 (+480691us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:52.574940 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:52.575367 (+ 427us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:52.575395 (+ 28us) write_op.cc:435] Acquired schema lock
0926 01:55:52.575408 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:52.613322 (+ 37914us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:52.613368 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:52.613391 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:52.613580 (+ 189us) tablet.cc:701] Row locks acquired
0926 01:55:52.613593 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:52.613733 (+ 140us) write_op.cc:270] Start()
0926 01:55:52.613824 (+ 91us) write_op.cc:276] Timestamp: P: 1790387752609535 usec, L: 0
0926 01:55:52.613841 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:55:52.614401 (+ 560us) log.cc:844] Serialized 405 byte log entry
0926 01:55:52.625372 (+ 10971us) op_driver.cc:464] REPLICATION: finished
0926 01:55:52.625737 (+ 365us) write_op.cc:301] APPLY: starting
0926 01:55:52.625797 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:52.625902 (+ 105us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:52.626003 (+ 101us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:52.626174 (+ 171us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:52.626305 (+ 131us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:52.626347 (+ 42us) write_op.cc:312] APPLY: finished
0926 01:55:52.626574 (+ 227us) log.cc:844] Serialized 23 byte log entry
0926 01:55:52.626732 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.626816 (+ 84us) write_op.cc:454] Released schema lock
0926 01:55:52.626850 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:52.627395 (+ 545us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.627419 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":1329,"apply.run_wall_time_us":1621,"num_ops":1,"prepare.queue_time_us":67543,"prepare.run_cpu_time_us":2333,"prepare.run_wall_time_us":42705,"raft.queue_time_us":88,"raft.run_cpu_time_us":659,"raft.run_wall_time_us":659,"replication_time_us":11430}]]}
I20260926 01:55:53.113976 9339 tablet_bootstrap.cc:654] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.120934 9703 raft_consensus.cc:515] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.128669 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:53.135314 9195 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.136071 9710 tablet_replica.cc:333] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:53.110664 8532 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_24ac8de493eb496bb7aeb552fef99c9c [id=b202eb85ec1944d8ad57f8521b37170b] alter complete (version 2)
I20260926 01:55:53.136921 9710 raft_consensus.cc:2243] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:53.137460 9710 raft_consensus.cc:2272] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:53.141201 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 01052a5e22a74a19919bae9dd0296291 with delete_type TABLET_DATA_DELETED (replaced by tablet e8e0e6ede1ce4487b08bfd5743e3656a) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.141532 9710 ts_tablet_manager.cc:1916] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:53.122406 8533 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 27) took 3983 ms (client timeout 1000 ms). Trace:
W20260926 01:55:53.142781 8533 rpcz_store.cc:269] 0926 01:55:49.139232 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.139441 (+ 209us) service_pool.cc:224] Handling call
0926 01:55:49.236869 (+ 97428us) spinlock_profiling.cc:243] Waited 95.4 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:53.122323 (+3885454us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27605,"reported_tablets":6,"spinlock_wait_cycles":219342080}
W20260926 01:55:53.145239 8536 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 39) took 1742 ms (client timeout 1000 ms). Trace:
W20260926 01:55:53.129475 8531 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 38) took 2739 ms (client timeout 1000 ms). Trace:
W20260926 01:55:53.145615 8536 rpcz_store.cc:269] 0926 01:55:51.403261 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:51.408697 (+ 5436us) service_pool.cc:224] Handling call
0926 01:55:53.145166 (+1736469us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33851,"reported_tablets":14}
I20260926 01:55:53.144471 8533 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_caaf07e0f7d24f7388b358362fd1668f" } modify_external_catalogs: true
I20260926 01:55:53.147758 8533 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_caaf07e0f7d24f7388b358362fd1668f" } modify_external_catalogs: true
I20260926 01:55:53.152503 9713 consensus_queue.cc:1048] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:55:53.157287 8534 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 37) took 2652 ms (client timeout 1000 ms). Trace:
I20260926 01:55:53.157995 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:53.146194 8531 rpcz_store.cc:269] 0926 01:55:50.390533 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:50.437436 (+ 46903us) service_pool.cc:224] Handling call
0926 01:55:53.129399 (+2691963us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:51.140315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:51.140745 (+ 430us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:51.140769 (+ 24us) write_op.cc:435] Acquired schema lock
0926 01:55:51.177845 (+ 37076us) tablet.cc:662] Decoding operations
0926 01:55:51.178215 (+ 370us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:51.178263 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:51.178288 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:51.178476 (+ 188us) tablet.cc:701] Row locks acquired
0926 01:55:51.178490 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:51.178645 (+ 155us) write_op.cc:270] Start()
0926 01:55:51.178746 (+ 101us) write_op.cc:276] Timestamp: P: 1790387751178576 usec, L: 0
0926 01:55:51.178760 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:51.179343 (+ 583us) log.cc:844] Serialized 530 byte log entry
0926 01:55:51.196718 (+ 17375us) op_driver.cc:464] REPLICATION: finished
0926 01:55:51.198388 (+ 1670us) write_op.cc:301] APPLY: starting
0926 01:55:51.198496 (+ 108us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:51.198605 (+ 109us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:51.198700 (+ 95us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:51.198881 (+ 181us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:51.199008 (+ 127us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:51.199056 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:55:51.199262 (+ 206us) log.cc:844] Serialized 23 byte log entry
0926 01:55:51.199422 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:51.199494 (+ 72us) write_op.cc:454] Released schema lock
0926 01:55:51.199527 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:51.238496 (+ 38969us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:51.238523 (+ 27us) write_op.cc:454] Released schema lock
Related trace 'op':
0926 01:55:51.540410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:51.540876 (+ 466us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:51.540900 (+ 24us) write_op.cc:435] Acquired schema lock
0926 01:55:51.540915 (+ 15us) tablet.cc:662] Decoding operations
0926 01:55:51.541199 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:51.541243 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:51.541266 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:51.541419 (+ 153us) tablet.cc:701] Row locks acquired
0926 01:55:51.541433 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:51.541554 (+ 121us) write_op.cc:270] Start()
0926 01:55:51.541654 (+ 100us) write_op.cc:276] Timestamp: P: 1790387751541497 usec, L: 0
0926 01:55:51.541668 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:51.542192 (+ 524us) log.cc:844] Serialized 356 byte log entry
0926 01:55:51.563407 (+ 21215us) op_driver.cc:464] REPLICATION: finished
0926 01:55:51.569367 (+ 5960us) write_op.cc:301] APPLY: starting
0926 01:55:51.569441 (+ 74us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:51.569541 (+ 100us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:51.569656 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:51.569849 (+ 193us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:51.570004 (+ 155us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:51.570047 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:55:51.570381 (+ 334us) log.cc:844] Serialized 23 byte log entry
0926 01:55:51.570605 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:51.570767 (+ 162us) write_op.cc:454] Released schema lock
0926 01:55:51.570808 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:51.571079 (+ 271us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:51.571103 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":288,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":1485,"apply.run_cpu_time_us":1361,"apply.run_wall_time_us":1361,"num_ops":1,"prepare.queue_time_us":216124,"prepare.run_cpu_time_us":2546,"prepare.run_wall_time_us":98116,"raft.queue_time_us":1284,"raft.run_cpu_time_us":734,"raft.run_wall_time_us":735,"replication_time_us":17846}],["op",{"apply.queue_time_us":5786,"apply.run_cpu_time_us":1526,"apply.run_wall_time_us":1667,"num_ops":1,"prepare.queue_time_us":331291,"prepare.run_cpu_time_us":1991,"prepare.run_wall_time_us":1991,"replication_time_us":21635}]]}
I20260926 01:55:53.172151 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b5bbdf5281dc45739e2ee7bf9aa6a02c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:53.173066 9339 tablet_bootstrap.cc:492] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.173530 9339 ts_tablet_manager.cc:1403] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.015s
I20260926 01:55:53.175444 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c6cccff3de64f3384ce58bb84c43487) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.176283 9339 raft_consensus.cc:359] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.176759 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.176920 9339 raft_consensus.cc:385] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.177182 9339 raft_consensus.cc:740] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.177459 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.177814 9339 consensus_queue.cc:260] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:53.151638 8538 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 28) took 3002 ms (client timeout 1000 ms). Trace:
W20260926 01:55:53.186335 8538 rpcz_store.cc:269] 0926 01:55:50.149152 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:50.165091 (+ 15939us) service_pool.cc:224] Handling call
0926 01:55:50.287030 (+121939us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:53.151545 (+2864515us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30993,"reported_tablets":10,"spinlock_wait_cycles":275251968}
I20260926 01:55:53.187475 8538 master_service.cc:960] ReplaceTablet: received request to replace tablet 03086ec5451743fdbf2d5726d43f31b1 from {username='slave'} at 127.0.0.1:42048
W20260926 01:55:53.142364 8540 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 38) took 1186 ms (client timeout 1000 ms). Trace:
I20260926 01:55:53.187893 9339 ts_tablet_manager.cc:1434] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
W20260926 01:55:53.188090 8540 rpcz_store.cc:269] 0926 01:55:51.955483 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:51.998344 (+ 42861us) service_pool.cc:224] Handling call
0926 01:55:53.142289 (+1143945us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31675,"reported_tablets":16}
I20260926 01:55:53.188771 9339 tablet_bootstrap.cc:492] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:53.189175 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_235a08a4f73a4761935c03af88c3a393" } new_table_name: "default.Table_3b74f2c3464144549a8ab60b6dd93279" modify_external_catalogs: true
W20260926 01:55:53.194780 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5bbdf5281dc45739e2ee7bf9aa6a02c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:53.177914 8532 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 29) took 2006 ms (client timeout 1000 ms). Trace:
I20260926 01:55:53.147444 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_a1fdc9d96e344c7bb928688dc2050a5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:53.197659 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1fdc9d96e344c7bb928688dc2050a5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:53.199303 9339 tablet_bootstrap.cc:654] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:53.198329 8532 rpcz_store.cc:269] 0926 01:55:51.171120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:51.313967 (+142847us) service_pool.cc:224] Handling call
0926 01:55:51.322392 (+ 8425us) spinlock_profiling.cc:243] Waited 5.76 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:53.177844 (+1855452us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:52.406181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:52.506252 (+100071us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:52.506281 (+ 29us) write_op.cc:435] Acquired schema lock
0926 01:55:52.506293 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:52.509405 (+ 3112us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:52.509454 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:52.509477 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:52.509664 (+ 187us) tablet.cc:701] Row locks acquired
0926 01:55:52.509678 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:52.509812 (+ 134us) write_op.cc:270] Start()
0926 01:55:52.509909 (+ 97us) write_op.cc:276] Timestamp: P: 1790387752509726 usec, L: 0
0926 01:55:52.509925 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:52.510706 (+ 781us) log.cc:844] Serialized 356 byte log entry
0926 01:55:52.523936 (+ 13230us) op_driver.cc:464] REPLICATION: finished
0926 01:55:52.536524 (+ 12588us) write_op.cc:301] APPLY: starting
0926 01:55:52.536596 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:52.536698 (+ 102us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:52.536786 (+ 88us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:52.536982 (+ 196us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:52.537138 (+ 156us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:52.537194 (+ 56us) write_op.cc:312] APPLY: finished
0926 01:55:52.537396 (+ 202us) log.cc:844] Serialized 23 byte log entry
0926 01:55:52.537572 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.537639 (+ 67us) write_op.cc:454] Released schema lock
0926 01:55:52.537674 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:52.537936 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.537960 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25791,"reported_tablets":15,"spinlock_wait_cycles":13242624,"child_traces":[["op",{"apply.queue_time_us":3042,"apply.run_cpu_time_us":1397,"apply.run_wall_time_us":1398,"num_ops":1,"prepare.queue_time_us":153340,"prepare.run_cpu_time_us":2277,"prepare.run_wall_time_us":104872,"raft.queue_time_us":2027,"raft.run_cpu_time_us":666,"raft.run_wall_time_us":665,"replication_time_us":13907}]]}
I20260926 01:55:53.214687 9339 tablet_bootstrap.cc:492] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.215142 9339 ts_tablet_manager.cc:1403] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.003s
I20260926 01:55:53.217832 9339 raft_consensus.cc:359] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.218513 9339 raft_consensus.cc:385] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.218771 9339 raft_consensus.cc:740] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
W20260926 01:55:53.133201 9003 leader_election.cc:343] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:53.138549 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.223131 9711 tablet_replica.cc:333] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:53.222729 9339 consensus_queue.cc:260] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.224149 9711 raft_consensus.cc:2243] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:53.138697 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet dfaca3fc02634f459083d23f3102f725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.225383 9339 ts_tablet_manager.cc:1434] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260926 01:55:53.140123 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b6daaf274894989b90da364f5da5250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.149972 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c6cccff3de64f3384ce58bb84c43487) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.150278 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.225406 9711 raft_consensus.cc:2272] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:53.228971 9711 ts_tablet_manager.cc:1916] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:53.158396 8534 rpcz_store.cc:269] 0926 01:55:50.504780 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:50.610587 (+105807us) service_pool.cc:224] Handling call
0926 01:55:53.157201 (+2546614us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:51.757731 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:51.758166 (+ 435us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:51.758190 (+ 24us) write_op.cc:435] Acquired schema lock
0926 01:55:51.758203 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:51.758475 (+ 272us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:51.758519 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:51.758542 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:51.758710 (+ 168us) tablet.cc:701] Row locks acquired
0926 01:55:51.758724 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:51.758846 (+ 122us) write_op.cc:270] Start()
0926 01:55:51.758932 (+ 86us) write_op.cc:276] Timestamp: P: 1790387751758768 usec, L: 0
0926 01:55:51.758949 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:55:51.759468 (+ 519us) log.cc:844] Serialized 530 byte log entry
0926 01:55:51.789995 (+ 30527us) op_driver.cc:464] REPLICATION: finished
0926 01:55:51.790966 (+ 971us) write_op.cc:301] APPLY: starting
0926 01:55:51.791038 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:51.791141 (+ 103us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:51.791247 (+ 106us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:51.791435 (+ 188us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:51.791567 (+ 132us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:51.791611 (+ 44us) write_op.cc:312] APPLY: finished
0926 01:55:51.791824 (+ 213us) log.cc:844] Serialized 23 byte log entry
0926 01:55:51.791994 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:51.792065 (+ 71us) write_op.cc:454] Released schema lock
0926 01:55:51.792100 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:51.792372 (+ 272us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:51.792396 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0926 01:55:52.511151 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:52.511608 (+ 457us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:52.511634 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:52.511646 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:52.526862 (+ 15216us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:52.526913 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:52.526942 (+ 29us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:52.527140 (+ 198us) tablet.cc:701] Row locks acquired
0926 01:55:52.527156 (+ 16us) write_op.cc:260] PREPARE: finished
0926 01:55:52.527304 (+ 148us) write_op.cc:270] Start()
0926 01:55:52.527411 (+ 107us) write_op.cc:276] Timestamp: P: 1790387752527215 usec, L: 0
0926 01:55:52.527431 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:55:52.544245 (+ 16814us) log.cc:844] Serialized 356 byte log entry
0926 01:55:52.568838 (+ 24593us) op_driver.cc:464] REPLICATION: finished
0926 01:55:52.571635 (+ 2797us) write_op.cc:301] APPLY: starting
0926 01:55:52.571728 (+ 93us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:52.571840 (+ 112us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:52.571932 (+ 92us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:52.572112 (+ 180us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:52.572252 (+ 140us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:52.572298 (+ 46us) write_op.cc:312] APPLY: finished
0926 01:55:52.572486 (+ 188us) log.cc:844] Serialized 23 byte log entry
0926 01:55:52.572656 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.572724 (+ 68us) write_op.cc:454] Released schema lock
0926 01:55:52.572758 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:52.573002 (+ 244us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.573027 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1158,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":786,"apply.run_cpu_time_us":1354,"apply.run_wall_time_us":1355,"num_ops":1,"prepare.queue_time_us":540334,"prepare.run_cpu_time_us":2094,"prepare.run_wall_time_us":3019,"raft.queue_time_us":157,"raft.run_cpu_time_us":677,"raft.run_wall_time_us":678,"replication_time_us":30939}],["op",{"apply.queue_time_us":2608,"apply.run_cpu_time_us":1340,"apply.run_wall_time_us":1339,"mutex_wait_us":142858,"num_ops":1,"prepare.queue_time_us":247429,"prepare.run_cpu_time_us":2330,"prepare.run_wall_time_us":33325,"replication_time_us":41300}]]}
W20260926 01:55:53.212237 8540 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 880d77d42c394e6a9cf0b3768a915402 consensus state has no leader
I20260926 01:55:53.231197 9339 tablet_bootstrap.cc:492] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:53.233456 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_bd491a7ccf524a75a6f05225b003cdc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:53.234011 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 100853us
W20260926 01:55:53.236442 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd491a7ccf524a75a6f05225b003cdc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:53.240190 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_cdcfa57f52a747aaa93bb04410df7061" } modify_external_catalogs: true
I20260926 01:55:53.240818 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_cdcfa57f52a747aaa93bb04410df7061" } modify_external_catalogs: true
I20260926 01:55:53.245055 9339 tablet_bootstrap.cc:654] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.248169 9809 tablet.cc:1751] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:53.246803 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_56df0fbe0b8949b4b6792b98d7fed21a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:53.250769 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56df0fbe0b8949b4b6792b98d7fed21a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:53.251374 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:53.251986 9219 raft_consensus.cc:3060] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.255354 9047 tablet_service.cc:1558] Processing DeleteTablet for tablet 01052a5e22a74a19919bae9dd0296291 with delete_type TABLET_DATA_DELETED (replaced by tablet e8e0e6ede1ce4487b08bfd5743e3656a) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.256068 9703 leader_election.cc:290] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:53.261158 9219 raft_consensus.cc:2468] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:53.263914 9340 tablet_bootstrap.cc:492] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:53.264497 9340 ts_tablet_manager.cc:1403] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.318s user 0.008s sys 0.008s
I20260926 01:55:53.269279 9339 tablet_bootstrap.cc:492] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.269717 9339 ts_tablet_manager.cc:1403] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.008s
I20260926 01:55:53.270884 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:53.274863 9339 raft_consensus.cc:359] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.275599 9339 raft_consensus.cc:385] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.275892 9339 raft_consensus.cc:740] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.276547 9339 consensus_queue.cc:260] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.278811 9339 ts_tablet_manager.cc:1434] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260926 01:55:53.281350 9340 raft_consensus.cc:359] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.282028 9339 tablet_bootstrap.cc:492] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:53.282132 9340 raft_consensus.cc:385] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.301307 8538 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 03086ec5451743fdbf2d5726d43f31b1
I20260926 01:55:53.302897 9339 tablet_bootstrap.cc:654] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.306702 8538 catalog_manager.cc:6934] ReplaceTablet: tablet 03086ec5451743fdbf2d5726d43f31b1 deleted and replaced by tablet 6ab082379974486bbea9f87f5aa64e10
I20260926 01:55:53.307250 8538 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 202) took 1001 ms. Trace:
W20260926 01:55:53.312994 9003 leader_election.cc:343] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:53.314697 9340 raft_consensus.cc:740] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.315522 9340 consensus_queue.cc:260] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.317027 9340 ts_tablet_manager.cc:1434] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20260926 01:55:53.318521 9340 tablet_bootstrap.cc:492] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:53.307619 8538 rpcz_store.cc:276] 0926 01:55:52.306358 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.187359 (+881001us) service_pool.cc:224] Handling call
0926 01:55:53.307189 (+119830us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:53.284192 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:53.284619 (+ 427us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:53.284645 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:53.284658 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:53.284974 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:53.285020 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:53.285042 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:53.285261 (+ 219us) tablet.cc:701] Row locks acquired
0926 01:55:53.285275 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:53.285406 (+ 131us) write_op.cc:270] Start()
0926 01:55:53.285496 (+ 90us) write_op.cc:276] Timestamp: P: 1790387753285359 usec, L: 0
0926 01:55:53.285512 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:53.286025 (+ 513us) log.cc:844] Serialized 659 byte log entry
0926 01:55:53.295880 (+ 9855us) op_driver.cc:464] REPLICATION: finished
0926 01:55:53.298237 (+ 2357us) write_op.cc:301] APPLY: starting
0926 01:55:53.298308 (+ 71us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:53.298439 (+ 131us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:53.298568 (+ 129us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:53.298945 (+ 377us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:53.299169 (+ 224us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:53.299211 (+ 42us) write_op.cc:312] APPLY: finished
0926 01:55:53.299425 (+ 214us) log.cc:844] Serialized 29 byte log entry
0926 01:55:53.299591 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.299671 (+ 80us) write_op.cc:454] Released schema lock
0926 01:55:53.299704 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:53.299947 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.299971 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2168,"apply.run_cpu_time_us":1686,"apply.run_wall_time_us":1686,"mutex_wait_us":180,"num_ops":2,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":2278,"prepare.run_wall_time_us":2426,"raft.queue_time_us":461,"raft.run_cpu_time_us":719,"raft.run_wall_time_us":721,"replication_time_us":10262}]]}
I20260926 01:55:53.324530 8538 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:44110 (request call id 17) took 2245 ms. Trace:
I20260926 01:55:53.324954 8538 rpcz_store.cc:276] 0926 01:55:51.078929 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.323885 (+2244956us) service_pool.cc:224] Handling call
0926 01:55:53.324463 (+ 578us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:53.336144 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet dfaca3fc02634f459083d23f3102f725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.345414 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab082379974486bbea9f87f5aa64e10) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.355893 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 23d2805b41d645d3bb905f26a380f383 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e12827ad93c47cf9013e238215ceb7a) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.359524 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a011a9edfe94e04afb6ea0367dd979f with delete_type TABLET_DATA_DELETED (replaced by tablet 17b8b0d54f5645e5963f1de217f94c84) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.378268 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_c30c84e91ae143e3b6918b9be1ec643e" } new_table_name: "default.Table_aa85c048a6eb4c18a82c25546cb20413" modify_external_catalogs: true
I20260926 01:55:53.379491 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 1866cd97bb024a9baecf101a6d5b1ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.383888 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b6daaf274894989b90da364f5da5250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.387581 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.389629 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet dccbfbc9fb5947f9b85ca8560e205b08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:44 UTC) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:53.381506 8537 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 30) took 1100 ms (client timeout 1000 ms). Trace:
W20260926 01:55:53.390945 8537 rpcz_store.cc:269] 0926 01:55:52.280952 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:52.318473 (+ 37521us) service_pool.cc:224] Handling call
0926 01:55:53.381426 (+1062953us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:52.714991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:52.715418 (+ 427us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:52.715444 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:52.715458 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:52.715750 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:52.715798 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:52.715821 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:52.715979 (+ 158us) tablet.cc:701] Row locks acquired
0926 01:55:52.715991 (+ 12us) write_op.cc:260] PREPARE: finished
0926 01:55:52.716134 (+ 143us) write_op.cc:270] Start()
0926 01:55:52.716222 (+ 88us) write_op.cc:276] Timestamp: P: 1790387752711929 usec, L: 0
0926 01:55:52.716238 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:52.716783 (+ 545us) log.cc:844] Serialized 530 byte log entry
0926 01:55:52.727492 (+ 10709us) op_driver.cc:464] REPLICATION: finished
0926 01:55:52.732471 (+ 4979us) write_op.cc:301] APPLY: starting
0926 01:55:52.732561 (+ 90us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:52.732672 (+ 111us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:52.732787 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:52.732972 (+ 185us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:52.733113 (+ 141us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:52.733158 (+ 45us) write_op.cc:312] APPLY: finished
0926 01:55:52.733365 (+ 207us) log.cc:844] Serialized 23 byte log entry
0926 01:55:52.733541 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.733612 (+ 71us) write_op.cc:454] Released schema lock
0926 01:55:52.733653 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:52.756458 (+ 22805us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:52.756486 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":50393,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":4776,"apply.run_cpu_time_us":1397,"apply.run_wall_time_us":1398,"num_ops":1,"prepare.queue_time_us":2465,"prepare.run_cpu_time_us":2149,"prepare.run_wall_time_us":41454,"raft.queue_time_us":112,"raft.run_cpu_time_us":730,"raft.run_wall_time_us":897,"replication_time_us":11154}]]}
I20260926 01:55:53.411129 8577 master_runner.cc:187] Error getting master registration for 127.8.76.188:38143: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111),
I20260926 01:55:53.411617 8577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:55:53.426802 9812 tablet_replica.cc:333] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:53.427878 9812 raft_consensus.cc:2243] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:53.428649 9812 pending_rounds.cc:70] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Trying to abort 1 pending ops.
I20260926 01:55:53.428936 9812 pending_rounds.cc:77] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428232067706880 op_type: NO_OP noop_request { }
I20260926 01:55:53.429410 9812 raft_consensus.cc:2889] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:53.305711 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab082379974486bbea9f87f5aa64e10) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.340353 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.341701 9195 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.358807 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:53.272714 9152 consensus_peers.cc:597] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:55:53.436734 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 164131us
I20260926 01:55:53.397457 9341 raft_consensus.cc:359] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.439957 9341 raft_consensus.cc:385] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.440250 9341 raft_consensus.cc:740] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.441063 9341 consensus_queue.cc:260] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.449121 9341 ts_tablet_manager.cc:1434] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.340s user 0.007s sys 0.001s
I20260926 01:55:53.449301 9006 leader_election.cc:304] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:53.453702 9703 raft_consensus.cc:2804] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:53.453913 9341 tablet_bootstrap.cc:492] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:53.454941 9703 raft_consensus.cc:697] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:53.455777 9703 consensus_queue.cc:237] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.474517 9339 tablet_bootstrap.cc:492] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.475085 9339 ts_tablet_manager.cc:1403] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.193s user 0.034s sys 0.039s
I20260926 01:55:53.477743 9339 raft_consensus.cc:359] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.482372 9339 raft_consensus.cc:385] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.482738 9339 raft_consensus.cc:740] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.477064 9711 ts_tablet_manager.cc:1929] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:53.483433 9711 log.cc:1199] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:53.483557 9339 consensus_queue.cc:260] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.485610 9339 ts_tablet_manager.cc:1434] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260926 01:55:53.489599 9341 tablet_bootstrap.cc:654] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.484246 9711 ts_tablet_manager.cc:1950] T 32e0fe60b8744dbb8ae2bafefe3e834d P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:53.492702 9594 consensus_queue.cc:1048] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:53.495119 9339 tablet_bootstrap.cc:492] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:53.508742 9339 tablet_bootstrap.cc:654] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.512006 9710 ts_tablet_manager.cc:1929] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260926 01:55:53.516412 9003 consensus_peers.cc:597] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:53.432504 8906 raft_consensus.cc:3060] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.434862 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 01052a5e22a74a19919bae9dd0296291 with delete_type TABLET_DATA_DELETED (replaced by tablet e8e0e6ede1ce4487b08bfd5743e3656a) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.518327 9770 raft_consensus.cc:493] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:53.436748 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab082379974486bbea9f87f5aa64e10) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.518812 9770 raft_consensus.cc:515] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.520859 9770 leader_election.cc:290] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:53.440610 8843 leader_election.cc:304] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:53.550258 9739 raft_consensus.cc:3060] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.553573 9339 tablet_bootstrap.cc:492] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.554016 9339 ts_tablet_manager.cc:1403] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.002s
I20260926 01:55:53.557022 9339 raft_consensus.cc:359] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:53.557708 9339 raft_consensus.cc:385] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.557955 9339 raft_consensus.cc:740] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.558446 9739 raft_consensus.cc:2749] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:53.558744 9339 consensus_queue.cc:260] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:53.559514 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 118861us
I20260926 01:55:53.560317 9339 ts_tablet_manager.cc:1434] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 01:55:53.562412 9340 tablet_bootstrap.cc:654] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.565204 9812 raft_consensus.cc:2272] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:53.567678 9812 ts_tablet_manager.cc:1916] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:53.383193 8536 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 200) took 1147 ms. Trace:
I20260926 01:55:53.425695 8531 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 201) took 1171 ms. Trace:
I20260926 01:55:53.576577 8536 rpcz_store.cc:276] 0926 01:55:52.236048 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.147037 (+910989us) service_pool.cc:224] Handling call
0926 01:55:53.198076 (+ 51039us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:55:53.343151 (+145075us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:55:53.382316 (+ 39165us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.382342 (+ 26us) write_op.cc:454] Released schema lock
0926 01:55:53.382580 (+ 238us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:55:53.382873 (+ 293us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:55:53.383131 (+ 258us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:53.344687 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:53.345120 (+ 433us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:53.345146 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:53.345159 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:53.345452 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:53.345498 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:53.345519 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:53.345728 (+ 209us) tablet.cc:701] Row locks acquired
0926 01:55:53.345741 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:53.345862 (+ 121us) write_op.cc:270] Start()
0926 01:55:53.345952 (+ 90us) write_op.cc:276] Timestamp: P: 1790387753345811 usec, L: 0
0926 01:55:53.345968 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:53.346533 (+ 565us) log.cc:844] Serialized 433 byte log entry
0926 01:55:53.354623 (+ 8090us) op_driver.cc:464] REPLICATION: finished
0926 01:55:53.355267 (+ 644us) write_op.cc:301] APPLY: starting
0926 01:55:53.355326 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:53.355434 (+ 108us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:53.355519 (+ 85us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:53.355971 (+ 452us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:53.356164 (+ 193us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:55:53.356208 (+ 44us) write_op.cc:312] APPLY: finished
0926 01:55:53.356512 (+ 304us) log.cc:844] Serialized 29 byte log entry
0926 01:55:53.356657 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.356726 (+ 69us) write_op.cc:454] Released schema lock
0926 01:55:53.356758 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":494,"apply.run_cpu_time_us":1530,"apply.run_wall_time_us":1692,"mutex_wait_us":176,"num_ops":2,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":2254,"prepare.run_wall_time_us":2439,"raft.queue_time_us":450,"raft.run_cpu_time_us":736,"raft.run_wall_time_us":737,"replication_time_us":8567}]]}
I20260926 01:55:53.434286 8534 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:55:53.577144 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_dd0ec322addf465e90e656d847756e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:53.435329 8538 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_24ac8de493eb496bb7aeb552fef99c9c" } new_table_name: "default.Table_c5aafc7dae5740a9bceb09dc9fa995c7" modify_external_catalogs: true
I20260926 01:55:53.582634 9711 tablet_replica.cc:333] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:53.583649 9711 raft_consensus.cc:2243] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:53.584218 9711 raft_consensus.cc:2272] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:53.495757 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 32e0fe60b8744dbb8ae2bafefe3e834d (table default.Table_17296161bc2343d1acf54e54e6e59e3b [id=8e48f349283d44a0a8bf7ae256529a4e]) successfully deleted
W20260926 01:55:53.585521 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 89784.3us
I20260926 01:55:53.586362 9711 ts_tablet_manager.cc:1916] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:53.587723 9339 tablet_bootstrap.cc:492] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:55:53.503036 8540 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 681900e0fac74cdabaea1d8a7fb4a189 consensus state has no leader
I20260926 01:55:53.589139 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.589701 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.589764 9047 tablet_service.cc:1558] Processing DeleteTablet for tablet 23d2805b41d645d3bb905f26a380f383 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e12827ad93c47cf9013e238215ceb7a) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:53.599045 8535 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 40) took 950 ms (client timeout 1000 ms). Trace:
W20260926 01:55:53.599448 8535 rpcz_store.cc:269] 0926 01:55:52.648200 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:52.697593 (+ 49393us) service_pool.cc:224] Handling call
0926 01:55:53.280578 (+582985us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.280611 (+ 33us) write_op.cc:454] Released schema lock
0926 01:55:53.598962 (+318351us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:53.115331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:53.115763 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:53.115790 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:55:53.115802 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:53.116073 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:53.116127 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:53.116149 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:53.116308 (+ 159us) tablet.cc:701] Row locks acquired
0926 01:55:53.116321 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:53.116445 (+ 124us) write_op.cc:270] Start()
0926 01:55:53.116535 (+ 90us) write_op.cc:276] Timestamp: P: 1790387753116409 usec, L: 0
0926 01:55:53.116550 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:53.117133 (+ 583us) log.cc:844] Serialized 530 byte log entry
0926 01:55:53.127077 (+ 9944us) op_driver.cc:464] REPLICATION: finished
0926 01:55:53.127499 (+ 422us) write_op.cc:301] APPLY: starting
0926 01:55:53.127565 (+ 66us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:53.127674 (+ 109us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:53.127786 (+ 112us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:53.127963 (+ 177us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:53.128104 (+ 141us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:53.128148 (+ 44us) write_op.cc:312] APPLY: finished
0926 01:55:53.128339 (+ 191us) log.cc:844] Serialized 23 byte log entry
0926 01:55:53.128487 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.128560 (+ 73us) write_op.cc:454] Released schema lock
0926 01:55:53.128596 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":254,"apply.run_cpu_time_us":1286,"apply.run_wall_time_us":1286,"num_ops":1,"prepare.queue_time_us":9177,"prepare.run_cpu_time_us":2161,"prepare.run_wall_time_us":2161,"raft.queue_time_us":2685,"raft.run_cpu_time_us":766,"raft.run_wall_time_us":768,"replication_time_us":10431}]]}
W20260926 01:55:53.602823 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd0ec322addf465e90e656d847756e56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:53.603787 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:53.614609 9003 leader_election.cc:343] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:53.522781 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:53.554492 9341 tablet_bootstrap.cc:492] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:53.622870 9341 ts_tablet_manager.cc:1403] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.171s user 0.030s sys 0.014s
I20260926 01:55:53.622790 9218 raft_consensus.cc:2468] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:53.619005 9808 raft_consensus.cc:493] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:53.623775 9808 raft_consensus.cc:515] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.625406 9341 raft_consensus.cc:359] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.625839 9808 leader_election.cc:290] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:53.626015 9341 raft_consensus.cc:385] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.626300 9341 raft_consensus.cc:740] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.626601 9710 log.cc:1199] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:53.626950 9341 consensus_queue.cc:260] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.628921 9341 ts_tablet_manager.cc:1434] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 01:55:53.629787 9341 tablet_bootstrap.cc:492] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:53.631006 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 681900e0fac74cdabaea1d8a7fb4a189 consensus state has no leader
I20260926 01:55:53.630859 9006 leader_election.cc:304] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:53.634876 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ef5265eb6244aa96c8195aa30cdfb0" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:53.635864 9341 tablet_bootstrap.cc:654] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.636664 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ef5265eb6244aa96c8195aa30cdfb0" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:53.637323 9073 raft_consensus.cc:2468] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:53.638823 9155 leader_election.cc:304] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:55:53.640604 9152 leader_election.cc:343] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:53.627475 9710 ts_tablet_manager.cc:1950] T 32e0fe60b8744dbb8ae2bafefe3e834d P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:53.645874 9808 raft_consensus.cc:2804] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:53.646232 9808 raft_consensus.cc:493] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:53.646588 9808 raft_consensus.cc:3060] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.647959 9340 tablet_bootstrap.cc:492] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:53.648651 9340 ts_tablet_manager.cc:1403] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.330s user 0.008s sys 0.003s
I20260926 01:55:53.577315 8531 rpcz_store.cc:276] 0926 01:55:52.254421 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.171752 (+917331us) service_pool.cc:224] Handling call
0926 01:55:53.195190 (+ 23438us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:55:53.253777 (+ 58587us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:55:53.425041 (+171264us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:55:53.425354 (+ 313us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:55:53.425625 (+ 271us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:53.409239 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:53.409661 (+ 422us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:53.409687 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:53.409701 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:53.409971 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:53.410009 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:53.410030 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:53.410228 (+ 198us) tablet.cc:701] Row locks acquired
0926 01:55:53.410242 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:53.410356 (+ 114us) write_op.cc:270] Start()
0926 01:55:53.410434 (+ 78us) write_op.cc:276] Timestamp: P: 1790387753410303 usec, L: 0
0926 01:55:53.410473 (+ 39us) op_driver.cc:348] REPLICATION: starting
0926 01:55:53.410985 (+ 512us) log.cc:844] Serialized 433 byte log entry
0926 01:55:53.420864 (+ 9879us) op_driver.cc:464] REPLICATION: finished
0926 01:55:53.421217 (+ 353us) write_op.cc:301] APPLY: starting
0926 01:55:53.421273 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:53.421389 (+ 116us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:53.421491 (+ 102us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:53.423922 (+ 2431us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:53.424062 (+ 140us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:55:53.424103 (+ 41us) write_op.cc:312] APPLY: finished
0926 01:55:53.424294 (+ 191us) log.cc:844] Serialized 29 byte log entry
0926 01:55:53.424439 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.424508 (+ 69us) write_op.cc:454] Released schema lock
0926 01:55:53.424542 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:53.424756 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.424785 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":1584,"apply.run_wall_time_us":3639,"mutex_wait_us":30094,"num_ops":2,"prepare.queue_time_us":30194,"prepare.run_cpu_time_us":3876,"prepare.run_wall_time_us":11205,"raft.queue_time_us":2073,"raft.run_cpu_time_us":633,"raft.run_wall_time_us":633,"replication_time_us":10299,"spinlock_wait_cycles":2816}]]}
I20260926 01:55:53.660027 9770 raft_consensus.cc:2804] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:53.660435 9770 raft_consensus.cc:493] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:53.660735 9770 raft_consensus.cc:3060] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.664104 9047 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.664791 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:53.665745 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 32e0fe60b8744dbb8ae2bafefe3e834d (table default.Table_17296161bc2343d1acf54e54e6e59e3b [id=8e48f349283d44a0a8bf7ae256529a4e]) successfully deleted
I20260926 01:55:53.666698 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:55:53.666978 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 01052a5e22a74a19919bae9dd0296291 because the tablet was not found. No further retry: Not found: Tablet not found: 01052a5e22a74a19919bae9dd0296291
W20260926 01:55:53.667152 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 51891.5us
W20260926 01:55:53.667608 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 1866cd97bb024a9baecf101a6d5b1ee4 because the tablet was not found. No further retry: Not found: Tablet not found: 1866cd97bb024a9baecf101a6d5b1ee4
I20260926 01:55:53.669000 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet 1df1d4daaddc4ac4aa02aa7763ea3184 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:53.669879 9770 raft_consensus.cc:515] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.675705 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:53.676313 9218 raft_consensus.cc:3060] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.683889 9218 raft_consensus.cc:2468] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:53.622331 9339 tablet_bootstrap.cc:654] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.688730 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:53.690776 9003 leader_election.cc:343] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:53.691393 9770 leader_election.cc:290] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:53.704298 9006 leader_election.cc:304] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:53.704768 9340 raft_consensus.cc:359] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.705715 9340 raft_consensus.cc:385] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.706020 9340 raft_consensus.cc:740] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.706696 9340 consensus_queue.cc:260] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.707571 9770 raft_consensus.cc:2804] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:53.708352 9770 raft_consensus.cc:697] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:53.709260 9770 consensus_queue.cc:237] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:53.713627 8504 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 56432.6us
I20260926 01:55:53.718755 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:53.722028 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 because the tablet was not found. No further retry: Not found: Tablet not found: 1df1d4daaddc4ac4aa02aa7763ea3184
I20260926 01:55:53.726552 9594 consensus_queue.cc:1048] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:53.734244 9710 tablet_replica.cc:333] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:53.747601 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_2bcec3148a5d418198f3c1adb758b949"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:53.749461 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bcec3148a5d418198f3c1adb758b949 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:53.752444 9339 tablet_bootstrap.cc:492] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:53.753144 9339 ts_tablet_manager.cc:1403] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.165s user 0.009s sys 0.000s
I20260926 01:55:53.759346 9341 tablet_bootstrap.cc:492] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:53.759837 9341 ts_tablet_manager.cc:1403] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.130s user 0.029s sys 0.028s
I20260926 01:55:53.760195 9710 raft_consensus.cc:2243] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:53.761234 9710 raft_consensus.cc:2272] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:53.790031 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_cef381857b034f04a16cd5bbd1f20eb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:53.790944 9340 ts_tablet_manager.cc:1434] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.142s user 0.006s sys 0.000s
W20260926 01:55:53.792014 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cef381857b034f04a16cd5bbd1f20eb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:53.792256 9340 tablet_bootstrap.cc:492] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:53.786926 9594 raft_consensus.cc:493] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:53.810896 9594 raft_consensus.cc:515] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.814224 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_632a5e0f37424fc9a71a1b75d1a4814c" } new_table_name: "default.Table_ed104ff312a24200ac22b165f4981c96" modify_external_catalogs: true
I20260926 01:55:53.833814 9339 raft_consensus.cc:359] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:53.834646 9339 raft_consensus.cc:385] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.834946 9339 raft_consensus.cc:740] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.838785 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:53.838580 9339 consensus_queue.cc:260] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:53.840757 8524 catalog_manager.cc:5136] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): alter failed for tablet a85e47c1bf9f4e96958aa2de41039180 (table default.Table_d4b013d8752449f898dd39794a37a01f [id=2c5fb89a77a6445e8aef61c7eaf0603f]),no further retry: Not found: Tablet not found: a85e47c1bf9f4e96958aa2de41039180
I20260926 01:55:53.762427 9341 raft_consensus.cc:359] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.845867 9341 raft_consensus.cc:385] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:53.846195 9341 raft_consensus.cc:740] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:53.846894 9341 consensus_queue.cc:260] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.853885 9594 leader_election.cc:290] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:53.872910 9594 raft_consensus.cc:493] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:53.873549 9594 raft_consensus.cc:515] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:53.749825 9808 raft_consensus.cc:515] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:53.875681 9594 leader_election.cc:290] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:53.881871 8539 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:53.883652 8539 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet b659eca3c97144aaacd41e747aecd96d (table default.Table_ee6cc37e9698474a8a3194f060b8c028 [id=0dae284524224264b7ccbd59cd975030]). Expected version 1 got 0
I20260926 01:55:53.884616 8539 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:53.885458 8533 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet a601d850375c43679dde7a46cfe71d22
I20260926 01:55:53.886876 8533 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 199) took 1685 ms. Trace:
I20260926 01:55:53.887179 8533 rpcz_store.cc:276] 0926 01:55:52.201718 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.144029 (+942311us) service_pool.cc:224] Handling call
0926 01:55:53.152264 (+ 8235us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:53.152819 (+ 555us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:53.152964 (+ 145us) catalog_manager.cc:2863] Locking tablets
0926 01:55:53.153524 (+ 560us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:53.195862 (+ 42338us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:53.196074 (+ 212us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:53.885368 (+689294us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:53.886813 (+ 1445us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:53.167094 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:53.167592 (+ 498us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:53.167621 (+ 29us) write_op.cc:435] Acquired schema lock
0926 01:55:53.167635 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:53.167995 (+ 360us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:53.168045 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:53.168072 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:53.168392 (+ 320us) tablet.cc:701] Row locks acquired
0926 01:55:53.168406 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:53.168543 (+ 137us) write_op.cc:270] Start()
0926 01:55:53.168651 (+ 108us) write_op.cc:276] Timestamp: P: 1790387753168502 usec, L: 0
0926 01:55:53.168666 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:53.169262 (+ 596us) log.cc:844] Serialized 762 byte log entry
0926 01:55:53.191922 (+ 22660us) op_driver.cc:464] REPLICATION: finished
0926 01:55:53.192493 (+ 571us) write_op.cc:301] APPLY: starting
0926 01:55:53.192555 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:53.192673 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:53.192772 (+ 99us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:53.193030 (+ 258us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:53.193180 (+ 150us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:53.193229 (+ 49us) write_op.cc:312] APPLY: finished
0926 01:55:53.193417 (+ 188us) log.cc:844] Serialized 29 byte log entry
0926 01:55:53.193574 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.193655 (+ 81us) write_op.cc:454] Released schema lock
0926 01:55:53.193693 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:53.193905 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:53.193928 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":414,"apply.run_cpu_time_us":1386,"apply.run_wall_time_us":1385,"num_ops":2,"prepare.queue_time_us":7236,"prepare.run_cpu_time_us":2552,"prepare.run_wall_time_us":3538,"raft.queue_time_us":343,"raft.run_cpu_time_us":696,"raft.run_wall_time_us":696,"replication_time_us":23162}]]}
I20260926 01:55:53.893608 9812 ts_tablet_manager.cc:1929] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:53.895131 8537 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d45fa6bcc6b14d1984b1e756819371a4" } modify_external_catalogs: true
I20260926 01:55:53.895660 8537 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d45fa6bcc6b14d1984b1e756819371a4" } modify_external_catalogs: true
I20260926 01:55:53.901605 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:53.904043 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:53.904374 9341 ts_tablet_manager.cc:1434] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.144s user 0.007s sys 0.000s
I20260926 01:55:53.905422 9341 tablet_bootstrap.cc:492] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:53.907248 8536 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4623456312284a698ae69d4054909647 consensus state has no leader
I20260926 01:55:53.918242 9339 ts_tablet_manager.cc:1434] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.165s user 0.005s sys 0.002s
I20260926 01:55:53.919503 9339 tablet_bootstrap.cc:492] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:53.921710 9196 tablet_service.cc:1511] Processing CreateTablet for tablet accc62f46ce24a4d97743dca05e65f51 (DEFAULT_TABLE table=default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.922803 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet accc62f46ce24a4d97743dca05e65f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:53.923058 9193 tablet_service.cc:1511] Processing CreateTablet for tablet 05042bcc0a6e426d9e3e8b25d8d55540 (DEFAULT_TABLE table=default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.924185 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05042bcc0a6e426d9e3e8b25d8d55540. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:53.895284 9828 log.cc:927] Time spent T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20260926 01:55:53.932286 9195 tablet_service.cc:1558] Processing DeleteTablet for tablet a601d850375c43679dde7a46cfe71d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:53.938815 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:53.939782 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:53.940349 9341 tablet_bootstrap.cc:654] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.942792 9710 ts_tablet_manager.cc:1916] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:53.964664 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:53.965468 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:55:53.971303 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ef5265eb6244aa96c8195aa30cdfb0" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:53.972076 9073 raft_consensus.cc:3060] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:53.977356 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 7e57300d2bcf4e0496ed13a4e7d967f3 (DEFAULT_TABLE table=default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:53.978425 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e57300d2bcf4e0496ed13a4e7d967f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:53.984263 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet b659eca3c97144aaacd41e747aecd96d (table default.Table_ee6cc37e9698474a8a3194f060b8c028 [id=0dae284524224264b7ccbd59cd975030]). Expected version 1 got 0
I20260926 01:55:53.987303 9711 ts_tablet_manager.cc:1929] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:53.987740 9711 log.cc:1199] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/dfaca3fc02634f459083d23f3102f725
I20260926 01:55:53.988448 9711 ts_tablet_manager.cc:1950] T dfaca3fc02634f459083d23f3102f725 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:53.992897 8532 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 70ef5265eb6244aa96c8195aa30cdfb0 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 4 got 0
I20260926 01:55:53.955739 9808 leader_election.cc:290] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:54.003765 9073 raft_consensus.cc:2468] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:54.004844 9339 tablet_bootstrap.cc:654] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:54.005173 9155 leader_election.cc:304] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:54.006175 9837 raft_consensus.cc:2804] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:54.006978 9837 raft_consensus.cc:697] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:54.013182 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:54.013904 9219 raft_consensus.cc:2468] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:54.016117 9703 raft_consensus.cc:493] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.017352 9049 tablet_service.cc:1511] Processing CreateTablet for tablet accc62f46ce24a4d97743dca05e65f51 (DEFAULT_TABLE table=default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.017581 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 05042bcc0a6e426d9e3e8b25d8d55540 (DEFAULT_TABLE table=default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.017724 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:54.018532 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet accc62f46ce24a4d97743dca05e65f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.018651 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet a601d850375c43679dde7a46cfe71d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:54.018779 9703 raft_consensus.cc:515] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.019369 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 7e57300d2bcf4e0496ed13a4e7d967f3 (DEFAULT_TABLE table=default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.020671 9703 leader_election.cc:290] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:54.021413 9051 tablet_service.cc:1511] Processing CreateTablet for tablet e76071c66b1d4635ac1a28110eb684fa (DEFAULT_TABLE table=default.Table_87c331af29e7419593789c486dc50a13 [id=2fa9e93da5704e3b9970e16f5afe93bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.022239 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e57300d2bcf4e0496ed13a4e7d967f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.022769 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05042bcc0a6e426d9e3e8b25d8d55540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.022735 9770 raft_consensus.cc:493] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.024538 9770 raft_consensus.cc:515] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:54.026719 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76071c66b1d4635ac1a28110eb684fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.040155 9812 log.cc:1199] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/dfaca3fc02634f459083d23f3102f725
I20260926 01:55:54.041083 9812 ts_tablet_manager.cc:1950] T dfaca3fc02634f459083d23f3102f725 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:53.893636 9340 tablet_bootstrap.cc:654] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:53.946866 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ef5265eb6244aa96c8195aa30cdfb0" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:54.048955 9198 tablet_service.cc:1511] Processing CreateTablet for tablet e76071c66b1d4635ac1a28110eb684fa (DEFAULT_TABLE table=default.Table_87c331af29e7419593789c486dc50a13 [id=2fa9e93da5704e3b9970e16f5afe93bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.050035 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76071c66b1d4635ac1a28110eb684fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:54.051208 9152 leader_election.cc:343] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:53.963292 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 75216.7us
I20260926 01:55:53.995033 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:54.003935 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:54.052820 9793 consensus_queue.cc:1048] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:54.065306 9837 consensus_queue.cc:237] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.071157 9812 tablet_replica.cc:333] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:54.072044 9812 raft_consensus.cc:2243] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:54.072669 9812 pending_rounds.cc:70] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Trying to abort 1 pending ops.
I20260926 01:55:54.072902 9812 pending_rounds.cc:77] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428234097324032 op_type: NO_OP noop_request { }
I20260926 01:55:54.073262 9812 raft_consensus.cc:2889] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:54.073598 9812 raft_consensus.cc:2272] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:54.079419 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:54.080382 8534 catalog_manager.cc:5838] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:54.084441 9340 tablet_bootstrap.cc:492] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:54.084838 9340 ts_tablet_manager.cc:1403] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.293s user 0.006s sys 0.012s
I20260926 01:55:54.087319 9340 raft_consensus.cc:359] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:54.087626 9006 consensus_peers.cc:597] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:54.087936 9340 raft_consensus.cc:385] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.088191 9340 raft_consensus.cc:740] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.089257 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:54.089118 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8552d28bce1c484c964fe1268e370cc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:54.090744 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8552d28bce1c484c964fe1268e370cc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.081554 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:54.093114 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:55:54.093726 8534 catalog_manager.cc:5838] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:54.094753 9340 consensus_queue.cc:260] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.095386 8534 catalog_manager.cc:5838] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:54.096724 9340 ts_tablet_manager.cc:1434] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260926 01:55:54.099617 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 70ef5265eb6244aa96c8195aa30cdfb0 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:54.103282 9812 ts_tablet_manager.cc:1916] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:54.110194 9793 raft_consensus.cc:493] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.110689 9793 raft_consensus.cc:515] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.112619 9793 leader_election.cc:290] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:54.120678 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d1175fab270f48b19ec70a1957bd6af6" } new_table_name: "default.Table_fe07ca51a9dd47b5a78a217875523300" modify_external_catalogs: true
I20260926 01:55:54.120630 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_ee8c0a5c849548228d58b9d2bb0c47ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:54.122377 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee8c0a5c849548228d58b9d2bb0c47ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.124709 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_59311caf53c74458b4c17a7907ed0afb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:54.126148 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59311caf53c74458b4c17a7907ed0afb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.128683 9340 tablet_bootstrap.cc:492] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:54.133196 8531 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_29ce250e9d9a4162966731d01a8e3b23" } modify_external_catalogs: true
I20260926 01:55:54.133738 8531 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_29ce250e9d9a4162966731d01a8e3b23" } modify_external_catalogs: true
I20260926 01:55:54.137291 9808 raft_consensus.cc:493] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.137790 9808 raft_consensus.cc:515] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:54.142592 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 88619.9us
I20260926 01:55:54.143011 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 70ef5265eb6244aa96c8195aa30cdfb0 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]). Expected version 4 got 0
I20260926 01:55:54.143572 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
I20260926 01:55:54.152488 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55452e70a7fe48efbafdc7cbb29c6272" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:54.153206 9341 tablet_bootstrap.cc:492] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:54.153703 9341 ts_tablet_manager.cc:1403] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.248s user 0.013s sys 0.000s
I20260926 01:55:54.156810 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 0c6cccff3de64f3384ce58bb84c43487 (DEFAULT_TABLE table=default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.157970 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6cccff3de64f3384ce58bb84c43487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.161536 8880 tablet_service.cc:1511] Processing CreateTablet for tablet accc62f46ce24a4d97743dca05e65f51 (DEFAULT_TABLE table=default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.162602 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet accc62f46ce24a4d97743dca05e65f51. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:54.164608 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bcb000f9234d4382ad019f63015b92e7 consensus state has no leader
I20260926 01:55:54.171780 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 05042bcc0a6e426d9e3e8b25d8d55540 (DEFAULT_TABLE table=default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.172890 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05042bcc0a6e426d9e3e8b25d8d55540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.190527 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 7e57300d2bcf4e0496ed13a4e7d967f3 (DEFAULT_TABLE table=default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.191648 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e57300d2bcf4e0496ed13a4e7d967f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.191320 9849 tablet.cc:1751] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:54.199128 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet a601d850375c43679dde7a46cfe71d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.201116 8879 tablet_service.cc:1511] Processing CreateTablet for tablet e76071c66b1d4635ac1a28110eb684fa (DEFAULT_TABLE table=default.Table_87c331af29e7419593789c486dc50a13 [id=2fa9e93da5704e3b9970e16f5afe93bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.202224 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76071c66b1d4635ac1a28110eb684fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:54.202361 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 97106.7us
W20260926 01:55:54.203626 9003 consensus_peers.cc:597] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:55:54.204550 9003 leader_election.cc:343] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:54.205436 9003 leader_election.cc:343] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:54.206707 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 94094.3us
I20260926 01:55:54.207655 9006 leader_election.cc:304] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:54.160234 9194 tablet_service.cc:1511] Processing CreateTablet for tablet b778c95327fc477ca210cb3c983665c1 (DEFAULT_TABLE table=default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.210157 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b778c95327fc477ca210cb3c983665c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.211163 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:54.212946 9152 leader_election.cc:343] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:54.162155 9197 tablet_service.cc:1511] Processing CreateTablet for tablet bcb000f9234d4382ad019f63015b92e7 (DEFAULT_TABLE table=default.Table_d1175fab270f48b19ec70a1957bd6af6 [id=88aedd8765374c6395d82b3752c327ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.215256 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcb000f9234d4382ad019f63015b92e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.215523 9847 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:54.172716 9196 tablet_service.cc:1511] Processing CreateTablet for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 (DEFAULT_TABLE table=default.Table_5bb843a35ff742499a87f56bb5d77d32 [id=760b382db3324c0a870435e2e4083241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.219182 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.223349 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 0c6cccff3de64f3384ce58bb84c43487 (DEFAULT_TABLE table=default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.224339 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6cccff3de64f3384ce58bb84c43487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.227877 9339 tablet_bootstrap.cc:492] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:54.228300 9339 ts_tablet_manager.cc:1403] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.309s user 0.007s sys 0.004s
I20260926 01:55:54.226496 9846 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:54.239154 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet dfaca3fc02634f459083d23f3102f725 (table default.Table_214d8f2884b048ebba2f07a67d5effe8 [id=bcf837a98f554432b1f529f1c6c5c4c0]) successfully deleted
I20260926 01:55:54.249969 9770 leader_election.cc:290] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:54.100577 9703 consensus_queue.cc:1048] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.145s
I20260926 01:55:54.252069 9340 tablet_bootstrap.cc:654] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:54.252285 9594 raft_consensus.cc:493] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.252774 9594 raft_consensus.cc:515] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.254694 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_3b74f2c3464144549a8ab60b6dd93279" } new_table_name: "default.Table_a6bc2356bd5e45899d0360430fe873e6" modify_external_catalogs: true
I20260926 01:55:54.255403 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:54.255960 9073 raft_consensus.cc:2468] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:54.257300 9155 leader_election.cc:304] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:54.254804 9829 raft_consensus.cc:2804] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:54.258168 9837 raft_consensus.cc:2804] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:54.258427 9829 raft_consensus.cc:493] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:54.258558 9837 raft_consensus.cc:493] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:54.258772 9829 raft_consensus.cc:3060] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:54.258868 9837 raft_consensus.cc:3060] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:54.263499 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 154) took 1125 ms. Trace:
I20260926 01:55:54.249046 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet b659eca3c97144aaacd41e747aecd96d (table default.Table_ee6cc37e9698474a8a3194f060b8c028 [id=0dae284524224264b7ccbd59cd975030]). Expected version 1 got 0
I20260926 01:55:54.263882 9711 rpcz_store.cc:276] 0926 01:55:53.138000 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.138565 (+ 565us) service_pool.cc:224] Handling call
0926 01:55:53.494343 (+355778us) ts_tablet_manager.cc:1151] Deleting tablet dfaca3fc02634f459083d23f3102f725
0926 01:55:53.494391 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:53.697778 (+203387us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:53.987131 (+289353us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.263417 (+276286us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18364,"tablet-delete.queue_time_us":268753}
I20260926 01:55:54.265275 9711 tablet_replica.cc:333] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:54.265646 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]). Expected version 1 got 0
I20260926 01:55:54.266136 9711 raft_consensus.cc:2243] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:54.266996 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_c5aafc7dae5740a9bceb09dc9fa995c7 [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 3 got 2
I20260926 01:55:54.269680 9339 raft_consensus.cc:359] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:54.278501 9339 raft_consensus.cc:385] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.285254 9339 raft_consensus.cc:740] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.287015 9856 tablet.cc:1751] T ac2e6e544aae4ac3a2d3cc116d22e581 P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:54.276296 9594 leader_election.cc:290] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:54.289511 8878 tablet_service.cc:1511] Processing CreateTablet for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 (DEFAULT_TABLE table=default.Table_5bb843a35ff742499a87f56bb5d77d32 [id=760b382db3324c0a870435e2e4083241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.290227 9837 raft_consensus.cc:515] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.290755 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.294349 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:54.294991 9073 raft_consensus.cc:3060] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:54.295667 9711 raft_consensus.cc:2272] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:54.297868 9711 ts_tablet_manager.cc:1916] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:54.301915 9073 raft_consensus.cc:2468] T ee418347ebb04927913810d539ef57ea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:54.268061 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:54.303751 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:54.304862 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 0c6cccff3de64f3384ce58bb84c43487
I20260926 01:55:54.306977 8880 tablet_service.cc:1511] Processing CreateTablet for tablet bcb000f9234d4382ad019f63015b92e7 (DEFAULT_TABLE table=default.Table_d1175fab270f48b19ec70a1957bd6af6 [id=88aedd8765374c6395d82b3752c327ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.308130 8881 tablet_service.cc:1511] Processing CreateTablet for tablet b778c95327fc477ca210cb3c983665c1 (DEFAULT_TABLE table=default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.308466 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcb000f9234d4382ad019f63015b92e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.309278 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b778c95327fc477ca210cb3c983665c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:54.311142 8539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 39) took 1332 ms (client timeout 1000 ms). Trace:
W20260926 01:55:54.311565 8539 rpcz_store.cc:269] 0926 01:55:52.978961 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.123496 (+144535us) service_pool.cc:224] Handling call
0926 01:55:54.311057 (+1187561us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50636,"reported_tablets":18}
I20260926 01:55:54.313614 9853 tablet.cc:1751] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:54.286973 9339 consensus_queue.cc:260] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:54.336335 9859 tablet.cc:1751] T ac2e6e544aae4ac3a2d3cc116d22e581 P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:54.338214 9710 ts_tablet_manager.cc:1929] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:54.338752 9710 log.cc:1199] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/f3bb1643da054583b97efd2e1a9e9c2f
I20260926 01:55:54.342024 9339 ts_tablet_manager.cc:1434] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.113s user 0.007s sys 0.000s
I20260926 01:55:54.343114 9339 tablet_bootstrap.cc:492] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:55:54.350347 9003 consensus_peers.cc:597] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:55:54.355053 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 148772us
I20260926 01:55:54.354434 9829 raft_consensus.cc:515] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.342998 9837 leader_election.cc:290] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:54.351184 9808 leader_election.cc:290] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:54.361225 9829 leader_election.cc:290] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:54.369302 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 118161us
I20260926 01:55:54.371114 9710 ts_tablet_manager.cc:1950] T f3bb1643da054583b97efd2e1a9e9c2f P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:54.373525 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 175) took 1197 ms. Trace:
W20260926 01:55:54.377653 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 61876.2us
I20260926 01:55:54.379082 9155 leader_election.cc:304] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:54.389681 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_42a00bb5a7f0447e9f894eb5d3019426"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:54.393133 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42a00bb5a7f0447e9f894eb5d3019426 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.399446 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_d724813e115e41aaa5a65a7f4b262a3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:54.400182 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_39a7b6f090bf44b5b0c15825bf0d75e5" } new_table_name: "default.Table_c123991aeaf149f4a35bd18938938414" modify_external_catalogs: true
W20260926 01:55:54.401244 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d724813e115e41aaa5a65a7f4b262a3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.373958 9710 rpcz_store.cc:276] 0926 01:55:53.176469 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.176607 (+ 138us) service_pool.cc:224] Handling call
0926 01:55:53.734039 (+557432us) ts_tablet_manager.cc:1151] Deleting tablet f3bb1643da054583b97efd2e1a9e9c2f
0926 01:55:53.734094 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.278018 (+543924us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.338064 (+ 60046us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.373449 (+ 35385us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":556886}
I20260926 01:55:54.396135 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c6cccff3de64f3384ce58bb84c43487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:54.419788 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet f3bb1643da054583b97efd2e1a9e9c2f (table default.Table_48399c44f8254aaeaa623db43553e1ca [id=eb2d6f7b9c9a48c49b34d0617267301e]) successfully deleted
I20260926 01:55:54.344879 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:54.424165 9152 leader_election.cc:343] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:54.427260 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 81331.1us
I20260926 01:55:54.427425 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:54.433084 9341 raft_consensus.cc:359] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.433851 9341 raft_consensus.cc:385] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.434289 9341 raft_consensus.cc:740] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:55:54.435148 9847 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.217s user 0.000s sys 0.000s
I20260926 01:55:54.435256 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet c27f1dd7bb0449a6a3fa371a8636e09b
I20260926 01:55:54.437237 8535 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 70ef5265eb6244aa96c8195aa30cdfb0
I20260926 01:55:54.438772 8535 catalog_manager.cc:6934] ReplaceTablet: tablet 70ef5265eb6244aa96c8195aa30cdfb0 deleted and replaced by tablet b612e8955d5f42dd8658320031fe36ca
I20260926 01:55:54.442612 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:54.443773 9340 tablet_bootstrap.cc:492] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:54.444208 9340 ts_tablet_manager.cc:1403] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.316s user 0.009s sys 0.000s
I20260926 01:55:54.445681 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 1df1d4daaddc4ac4aa02aa7763ea3184 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c6cccff3de64f3384ce58bb84c43487) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.448479 9340 raft_consensus.cc:359] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:54.458387 8532 catalog_manager.cc:6038] Failed to send AlterTable RPC for tablet 70ef5265eb6244aa96c8195aa30cdfb0 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 70ef5265eb6244aa96c8195aa30cdfb0 consensus state has no leader
W20260926 01:55:54.462438 8532 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 31) took 997 ms (client timeout 1000 ms). Trace:
W20260926 01:55:54.462899 8532 rpcz_store.cc:269] 0926 01:55:53.465287 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.545411 (+ 80124us) service_pool.cc:224] Handling call
0926 01:55:54.462331 (+916920us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47933,"reported_tablets":21}
I20260926 01:55:54.466272 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 with delete_type TABLET_DATA_DELETED (replaced by tablet b612e8955d5f42dd8658320031fe36ca) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:54.466748 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:54.467784 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 32e0fe60b8744dbb8ae2bafefe3e834d with delete_type TABLET_DATA_DELETED (replaced by tablet cb57d9fd279344bf8245e7fc2cd5e54b) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:54.467784 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet f3bb1643da054583b97efd2e1a9e9c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:54.470317 8534 catalog_manager.cc:6038] Failed to send AlterTable RPC for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 3571967749844fd7859cf179a93cf403 consensus state has no leader
W20260926 01:55:54.472536 8534 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 41) took 777 ms (client timeout 1000 ms). Trace:
I20260926 01:55:54.450110 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.457547 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 with delete_type TABLET_DATA_DELETED (replaced by tablet b612e8955d5f42dd8658320031fe36ca) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.467549 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bac55da5b164c0f981914d39c87c33e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:54.482988 9340 raft_consensus.cc:385] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.468089 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c6cccff3de64f3384ce58bb84c43487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.468506 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.475878 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet dfaca3fc02634f459083d23f3102f725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.483304 9340 raft_consensus.cc:740] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.484650 9340 consensus_queue.cc:260] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.488157 9340 ts_tablet_manager.cc:1434] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260926 01:55:54.490984 9340 tablet_bootstrap.cc:492] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:54.507396 8532 master_service.cc:960] ReplaceTablet: received request to replace tablet 9bac55da5b164c0f981914d39c87c33e from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:54.507397 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55452e70a7fe48efbafdc7cbb29c6272" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:54.523600 9339 tablet_bootstrap.cc:700] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2: Time spent opening tablet: real 0.144s user 0.001s sys 0.000s
I20260926 01:55:54.524400 9339 tablet_bootstrap.cc:654] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:54.472882 8534 rpcz_store.cc:269] 0926 01:55:53.695096 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.695249 (+ 153us) service_pool.cc:224] Handling call
0926 01:55:54.472454 (+777205us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:54.165863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:54.166211 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:54.166236 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:55:54.166249 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:54.166855 (+ 606us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:54.166893 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:54.166917 (+ 24us) tablet.cc:685] Acquiring locks for 3 operations
0926 01:55:54.167164 (+ 247us) tablet.cc:701] Row locks acquired
0926 01:55:54.167177 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:54.167291 (+ 114us) write_op.cc:270] Start()
0926 01:55:54.167371 (+ 80us) write_op.cc:276] Timestamp: P: 1790387754167242 usec, L: 0
0926 01:55:54.167385 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:54.167878 (+ 493us) log.cc:844] Serialized 1284 byte log entry
0926 01:55:54.195620 (+ 27742us) op_driver.cc:464] REPLICATION: finished
0926 01:55:54.196928 (+ 1308us) write_op.cc:301] APPLY: starting
0926 01:55:54.196992 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:54.197126 (+ 134us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:54.197236 (+ 110us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:54.197562 (+ 326us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:54.197704 (+ 142us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0926 01:55:54.197746 (+ 42us) write_op.cc:312] APPLY: finished
0926 01:55:54.197952 (+ 206us) log.cc:844] Serialized 35 byte log entry
0926 01:55:54.198102 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:54.198181 (+ 79us) write_op.cc:454] Released schema lock
0926 01:55:54.198217 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:54.198516 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:54.198550 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":1100,"apply.run_cpu_time_us":1501,"apply.run_wall_time_us":1501,"num_ops":3,"prepare.queue_time_us":2236,"prepare.run_cpu_time_us":2087,"prepare.run_wall_time_us":3256,"replication_time_us":28125}]]}
I20260926 01:55:54.528818 9710 tablet_replica.cc:333] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:54.529806 9710 raft_consensus.cc:2243] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:54.534601 9805 raft_consensus.cc:2804] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:54.535116 9805 raft_consensus.cc:697] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:54.535897 9805 consensus_queue.cc:237] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.538735 9710 raft_consensus.cc:2272] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:54.539572 9341 consensus_queue.cc:260] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.540934 9710 ts_tablet_manager.cc:1916] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:54.542049 9341 ts_tablet_manager.cc:1434] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.388s user 0.010s sys 0.000s
I20260926 01:55:54.548265 8843 leader_election.cc:304] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:54.549482 9793 raft_consensus.cc:2749] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:54.550246 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:54.553330 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 174349us
I20260926 01:55:54.556524 9812 ts_tablet_manager.cc:1929] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:54.556978 9812 log.cc:1199] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/23d2805b41d645d3bb905f26a380f383
I20260926 01:55:54.557775 9812 ts_tablet_manager.cc:1950] T 23d2805b41d645d3bb905f26a380f383 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:54.568171 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 165) took 1231 ms. Trace:
I20260926 01:55:54.568596 9812 rpcz_store.cc:276] 0926 01:55:53.337121 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.355728 (+ 18607us) service_pool.cc:224] Handling call
0926 01:55:54.070968 (+715240us) ts_tablet_manager.cc:1151] Deleting tablet 23d2805b41d645d3bb905f26a380f383
0926 01:55:54.071020 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.325420 (+254400us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.556358 (+230938us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.568095 (+ 11737us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":714602}
I20260926 01:55:54.570204 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 23d2805b41d645d3bb905f26a380f383 (table default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]) successfully deleted
I20260926 01:55:54.574994 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 166) took 1218 ms. Trace:
I20260926 01:55:54.575388 9812 rpcz_store.cc:276] 0926 01:55:53.356771 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.359356 (+ 2585us) service_pool.cc:224] Handling call
0926 01:55:54.574677 (+1215321us) ts_tablet_manager.cc:1151] Deleting tablet 8a011a9edfe94e04afb6ea0367dd979f
0926 01:55:54.574726 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.574919 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1214742}
W20260926 01:55:54.576869 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 8a011a9edfe94e04afb6ea0367dd979f because the tablet was not found. No further retry: Not found: Tablet not found: 8a011a9edfe94e04afb6ea0367dd979f
W20260926 01:55:54.581099 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 373574us
W20260926 01:55:54.581816 9006 leader_election.cc:343] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:54.582217 9006 leader_election.cc:304] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:54.586804 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_f820adee11bf4557b6119642992da842" } modify_external_catalogs: true
I20260926 01:55:54.587466 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_f820adee11bf4557b6119642992da842" } modify_external_catalogs: true
I20260926 01:55:54.597970 9340 tablet_bootstrap.cc:654] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:54.599737 9339 tablet_bootstrap.cc:492] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:54.603580 9339 ts_tablet_manager.cc:1403] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.261s user 0.011s sys 0.001s
I20260926 01:55:54.603752 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 167) took 1230 ms. Trace:
W20260926 01:55:54.609100 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 880d77d42c394e6a9cf0b3768a915402 consensus state has no leader
I20260926 01:55:54.614560 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 23d2805b41d645d3bb905f26a380f383 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e12827ad93c47cf9013e238215ceb7a) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.604136 9812 rpcz_store.cc:276] 0926 01:55:53.373636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.379291 (+ 5655us) service_pool.cc:224] Handling call
0926 01:55:54.603337 (+1224046us) ts_tablet_manager.cc:1151] Deleting tablet 1866cd97bb024a9baecf101a6d5b1ee4
0926 01:55:54.603392 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.603664 (+ 272us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223341}
W20260926 01:55:54.616894 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 1866cd97bb024a9baecf101a6d5b1ee4 because the tablet was not found. No further retry: Not found: Tablet not found: 1866cd97bb024a9baecf101a6d5b1ee4
I20260926 01:55:54.618902 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.621212 9812 tablet_replica.cc:333] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:54.601079 9867 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:55:54.631493 9812 raft_consensus.cc:2243] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 01:55:54.635166 9862 log.cc:927] Time spent T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20260926 01:55:54.630298 9849 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58226 (request call id 163) took 1294 ms. Trace:
W20260926 01:55:54.641796 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 70ef5265eb6244aa96c8195aa30cdfb0 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 70ef5265eb6244aa96c8195aa30cdfb0 consensus state has no leader
I20260926 01:55:54.641453 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_6785898585374a828eb715d1bb3fef65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:54.642805 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b9e652042f524ac7973179d178f9fb52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:54.636942 9849 rpcz_store.cc:276] 0926 01:55:53.335599 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.348256 (+ 12657us) service_pool.cc:224] Handling call
0926 01:55:54.630216 (+1281960us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:53.360152 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:55:53.360488 (+ 336us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:55:53.360517 (+ 29us) alter_schema_op.cc:66] Acquired schema lock
0926 01:55:53.360559 (+ 42us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:55:53.360693 (+ 134us) alter_schema_op.cc:128] START. Timestamp: 1790387753378486
0926 01:55:53.360713 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:55:53.361213 (+ 500us) log.cc:844] Serialized 173 byte log entry
0926 01:55:53.507363 (+146150us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b54000afda0. stack: 00007f94aec19323 00007f94aec19220 00007f94ae7bd574 0000000000516964 00000000005168f9 00007f94b5d175c0 00007f94b5d24be8 00007f94b5d2482a 00007f94b5da9125 00007f94b5da95c9 00007f94b6239791 00007f94b6238f68 00007f94b623dc01 00007f94b623dbb9 00007f94b623db49 00007f94b623db11
0926 01:55:54.146138 (+638775us) op_driver.cc:464] REPLICATION: finished
0926 01:55:54.155201 (+ 9063us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:55:54.193479 (+ 38278us) log.cc:844] Serialized 5 byte log entry
0926 01:55:54.602484 (+409005us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.629940 (+ 27456us) log.cc:844] Serialized 14 byte log entry
0926 01:55:54.630048 (+ 108us) alter_schema_op.cc:73] Released schema lock
0926 01:55:54.630079 (+ 31us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"spinlock_wait_cycles":23808,"child_traces":[["op",{"apply.queue_time_us":8877,"mutex_wait_us":23454,"prepare.queue_time_us":10919,"prepare.run_cpu_time_us":14005,"prepare.run_wall_time_us":147337,"replication_time_us":785295,"spinlock_wait_cycles":55786368,"thread_start_us":214162,"threads_started":2,"wal-append.queue_time_us":58510}]]}
I20260926 01:55:54.640079 9812 raft_consensus.cc:2272] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:54.647748 9812 ts_tablet_manager.cc:1916] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:54.642529 9340 tablet_bootstrap.cc:492] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:54.650772 9340 ts_tablet_manager.cc:1403] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.160s user 0.009s sys 0.002s
I20260926 01:55:54.653266 9340 raft_consensus.cc:359] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.653900 9340 raft_consensus.cc:385] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.654158 9340 raft_consensus.cc:740] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.654953 9340 consensus_queue.cc:260] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.658733 9829 raft_consensus.cc:2749] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:55:54.659408 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 83352.5us
I20260926 01:55:54.660267 9340 ts_tablet_manager.cc:1434] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
W20260926 01:55:54.660359 9003 leader_election.cc:343] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:54.661033 9003 consensus_peers.cc:597] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:55:54.661522 9003 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:54.662385 9829 consensus_queue.cc:1048] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:54.662829 8537 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_fb988b16dc1843a7b22bced528ea5ea6" } modify_external_catalogs: true
I20260926 01:55:54.664012 8537 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_fb988b16dc1843a7b22bced528ea5ea6" } modify_external_catalogs: true
W20260926 01:55:54.665360 9003 leader_election.cc:343] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:54.668062 9003 leader_election.cc:343] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:54.668512 8536 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8e0e6ede1ce4487b08bfd5743e3656a consensus state has no leader
W20260926 01:55:54.677915 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9e652042f524ac7973179d178f9fb52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:54.678951 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6785898585374a828eb715d1bb3fef65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.681229 9339 raft_consensus.cc:359] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:54.681928 9339 raft_consensus.cc:385] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.682188 9339 raft_consensus.cc:740] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.685765 9711 ts_tablet_manager.cc:1929] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:54.687150 9340 tablet_bootstrap.cc:492] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:54.701207 9805 raft_consensus.cc:493] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.702157 9711 log.cc:1199] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/3b6daaf274894989b90da364f5da5250
I20260926 01:55:54.701893 9805 raft_consensus.cc:515] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.706147 9341 tablet_bootstrap.cc:492] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:54.710403 9340 tablet_bootstrap.cc:654] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:54.712406 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:54.713869 9152 leader_election.cc:343] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:54.717005 9341 tablet_bootstrap.cc:654] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:54.705040 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:54.727550 9069 raft_consensus.cc:2468] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:54.729213 9155 leader_election.cc:304] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:54.730301 9858 raft_consensus.cc:2804] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:54.730685 9858 raft_consensus.cc:493] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:54.730983 9858 raft_consensus.cc:3060] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:54.736172 9805 leader_election.cc:290] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:54.738744 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet 17b8b0d54f5645e5963f1de217f94c84 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:54.740417 9340 tablet_bootstrap.cc:492] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:54.740899 9340 ts_tablet_manager.cc:1403] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.005s
I20260926 01:55:54.743650 9340 raft_consensus.cc:359] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.744326 9340 raft_consensus.cc:385] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.744585 9340 raft_consensus.cc:740] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.745390 9340 consensus_queue.cc:260] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.749415 9710 ts_tablet_manager.cc:1929] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:54.749857 9710 log.cc:1199] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/75d4598596ff49378fc3536fff0ac1b1
I20260926 01:55:54.751416 9340 ts_tablet_manager.cc:1434] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260926 01:55:54.752429 9858 raft_consensus.cc:515] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.756366 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:54.757009 9069 raft_consensus.cc:3060] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:54.759747 9340 tablet_bootstrap.cc:492] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:54.764884 9069 raft_consensus.cc:2468] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:54.768151 9710 ts_tablet_manager.cc:1950] T 75d4598596ff49378fc3536fff0ac1b1 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:54.770495 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 176) took 1593 ms. Trace:
I20260926 01:55:54.770910 9710 rpcz_store.cc:276] 0926 01:55:53.177172 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.177335 (+ 163us) service_pool.cc:224] Handling call
0926 01:55:54.419114 (+1241779us) ts_tablet_manager.cc:1151] Deleting tablet 75d4598596ff49378fc3536fff0ac1b1
0926 01:55:54.528646 (+109532us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.673107 (+144461us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.749252 (+ 76145us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.770394 (+ 21142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8359,"tablet-delete.queue_time_us":1241290}
I20260926 01:55:54.773102 9710 tablet_replica.cc:333] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:54.773376 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 75d4598596ff49378fc3536fff0ac1b1 (table default.Table_8fa5adfde54e424d935ea9df94631389 [id=d706c478a4b54165a5e8943317180616]) successfully deleted
I20260926 01:55:54.773854 9710 raft_consensus.cc:2243] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:54.774379 9710 raft_consensus.cc:2272] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:54.775658 9340 tablet_bootstrap.cc:654] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:54.780969 9793 raft_consensus.cc:493] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 01:55:54.686873 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 105083us
I20260926 01:55:54.781514 9793 raft_consensus.cc:515] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:54.782335 9006 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:54.783638 9793 leader_election.cc:290] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:54.783869 9155 leader_election.cc:304] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:54.784654 9805 raft_consensus.cc:2804] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:54.785097 9805 raft_consensus.cc:697] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:54.786003 9710 ts_tablet_manager.cc:1916] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:54.785869 9805 consensus_queue.cc:237] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.787334 9340 tablet_bootstrap.cc:492] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:54.787747 9340 ts_tablet_manager.cc:1403] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.008s
I20260926 01:55:54.790416 9340 raft_consensus.cc:359] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.790745 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:54.791066 9340 raft_consensus.cc:385] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.791401 9218 raft_consensus.cc:2393] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:54.791294 9340 raft_consensus.cc:740] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.792109 9340 consensus_queue.cc:260] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.801447 9812 ts_tablet_manager.cc:1929] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:54.802007 9812 log.cc:1199] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/3b6daaf274894989b90da364f5da5250
I20260926 01:55:54.802932 9812 ts_tablet_manager.cc:1950] T 3b6daaf274894989b90da364f5da5250 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:54.810190 9006 leader_election.cc:304] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:54.767390 9711 ts_tablet_manager.cc:1950] T 3b6daaf274894989b90da364f5da5250 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:54.726511 9339 consensus_queue.cc:260] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:54.818783 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_776639f9efaf43db9a1ca82994fe91b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:54.820446 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_776639f9efaf43db9a1ca82994fe91b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.804968 8902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:54.822337 9152 leader_election.cc:343] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:54.806846 9340 ts_tablet_manager.cc:1434] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.019s user 0.004s sys 0.003s
I20260926 01:55:54.811208 9793 raft_consensus.cc:493] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.823506 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 168) took 1443 ms. Trace:
I20260926 01:55:54.823312 9793 raft_consensus.cc:515] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.823894 9812 rpcz_store.cc:276] 0926 01:55:53.380435 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.383710 (+ 3275us) service_pool.cc:224] Handling call
0926 01:55:54.620995 (+1237285us) ts_tablet_manager.cc:1151] Deleting tablet 3b6daaf274894989b90da364f5da5250
0926 01:55:54.621050 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.680832 (+ 59782us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.801297 (+120465us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.823438 (+ 22141us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236151}
I20260926 01:55:54.825304 9793 leader_election.cc:290] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:54.826910 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:54.826818 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_e3f193c692be4f919e17e315a529b202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:54.828383 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3f193c692be4f919e17e315a529b202 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.830890 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_84d6c2cf62bb42779b018cfc68f2b42f" } new_table_name: "default.Table_f76e778fb54a460b89d85239e3fe34dd" modify_external_catalogs: true
I20260926 01:55:54.831552 9340 tablet_bootstrap.cc:492] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:54.835371 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 3b6daaf274894989b90da364f5da5250 (table default.Table_efe5a3dc3242481484fedd388b7be098 [id=be02dac770c84af69e2ff172417a074a]) successfully deleted
I20260926 01:55:54.839895 9808 raft_consensus.cc:493] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:54.840401 9808 raft_consensus.cc:515] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.844259 9808 leader_election.cc:290] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:54.825277 9812 tablet_replica.cc:333] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:54.861979 8902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:54.865833 9152 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:54.863269 9812 raft_consensus.cc:2243] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:54.874660 9812 raft_consensus.cc:2272] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:54.878012 9812 ts_tablet_manager.cc:1916] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:54.879484 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 5cbf646569e6497dac90e308f0681372 (DEFAULT_TABLE table=default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.894400 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbf646569e6497dac90e308f0681372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.899942 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:54.900833 9069 raft_consensus.cc:2468] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:54.902228 9155 leader_election.cc:304] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:54.903441 9808 raft_consensus.cc:2804] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:54.903832 9808 raft_consensus.cc:493] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:54.904265 9808 raft_consensus.cc:3060] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:54.911656 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 5eef6a48e87e4c61a23058942a82bccc (DEFAULT_TABLE table=default.Table_b5bbdf5281dc45739e2ee7bf9aa6a02c [id=37400e3a991842198609512f0a9aeead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.912833 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eef6a48e87e4c61a23058942a82bccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:54.920959 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_6ba9495c267245aa86a1b9184b17a7f1" } new_table_name: "default.Table_4eb6a42428994acda28b9ba1bf7d5256" modify_external_catalogs: true
I20260926 01:55:54.925495 9858 leader_election.cc:290] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:54.936081 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:55:54.938700 8531 catalog_manager.cc:6934] ReplaceTablet: tablet 17b8b0d54f5645e5963f1de217f94c84 deleted and replaced by tablet 6752df6a58914df5b8718dd0439aa885
I20260926 01:55:54.938941 9340 tablet_bootstrap.cc:654] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:54.940227 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 0d068e93fd374a6091d7473db069a7fa
I20260926 01:55:54.945024 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 6752df6a58914df5b8718dd0439aa885) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.945704 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 0d068e93fd374a6091d7473db069a7fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:54.959175 8537 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b644d2630c07480e8cc4ec135c1b91f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:54.961082 8537 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b644d2630c07480e8cc4ec135c1b91f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:54.961539 9340 tablet_bootstrap.cc:492] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:54.961994 9340 ts_tablet_manager.cc:1403] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.131s user 0.008s sys 0.003s
I20260926 01:55:54.965102 9340 raft_consensus.cc:359] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:54.965731 9340 raft_consensus.cc:385] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:54.965974 9340 raft_consensus.cc:740] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:54.969842 9703 raft_consensus.cc:2749] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:54.971719 9339 ts_tablet_manager.cc:1434] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.346s user 0.006s sys 0.000s
W20260926 01:55:54.974356 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 192052us
I20260926 01:55:54.976029 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 153) took 1838 ms. Trace:
W20260926 01:55:54.962424 8539 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f10cb68586b4a73abb86641b90c12f7 consensus state has no leader
I20260926 01:55:54.976976 9711 rpcz_store.cc:276] 0926 01:55:53.137721 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.139954 (+ 2233us) service_pool.cc:224] Handling call
0926 01:55:54.265092 (+1125138us) ts_tablet_manager.cc:1151] Deleting tablet 3b6daaf274894989b90da364f5da5250
0926 01:55:54.265143 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.516597 (+251454us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.644943 (+128346us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:54.975956 (+331013us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":154780,"tablet-delete.queue_time_us":1039270}
I20260926 01:55:54.964135 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_bbbb170151dd4078915478e24d1c1ee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:54.979008 9872 tablet.cc:1751] T ac2e6e544aae4ac3a2d3cc116d22e581 P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260926 01:55:54.979887 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbbb170151dd4078915478e24d1c1ee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:54.983614 9152 consensus_peers.cc:597] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:55:54.984920 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 54789.7us
I20260926 01:55:54.991289 9339 tablet_bootstrap.cc:492] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:54.998256 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bac55da5b164c0f981914d39c87c33e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:54.999862 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet dfaca3fc02634f459083d23f3102f725 (table default.Table_214d8f2884b048ebba2f07a67d5effe8 [id=bcf837a98f554432b1f529f1c6c5c4c0]) successfully deleted
I20260926 01:55:55.004369 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 156) took 1854 ms. Trace:
I20260926 01:55:55.006166 9711 rpcz_store.cc:276] 0926 01:55:53.149719 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.149834 (+ 115us) service_pool.cc:224] Handling call
0926 01:55:54.978656 (+1828822us) ts_tablet_manager.cc:1151] Deleting tablet 1df1d4daaddc4ac4aa02aa7763ea3184
0926 01:55:55.004085 (+ 25429us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.004298 (+ 213us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1752784}
I20260926 01:55:55.012197 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:55.008946 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cba678b66744dcfb4d13ae07ebec33f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:55.015740 9711 tablet_replica.cc:333] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:55.022050 9711 raft_consensus.cc:2243] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:55.025533 8843 leader_election.cc:304] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:55.027199 9843 raft_consensus.cc:2749] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:55.030285 8902 raft_consensus.cc:3060] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.030300 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:55.030920 9219 raft_consensus.cc:3060] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.038154 9219 raft_consensus.cc:2468] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:55:55.022985 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c36ea09cf15647c0a344c9ba9b289d61" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:55.040944 9711 raft_consensus.cc:2272] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:55.045644 9808 raft_consensus.cc:515] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.048453 9808 leader_election.cc:290] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:55.052970 9341 tablet_bootstrap.cc:492] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:55.053433 9341 ts_tablet_manager.cc:1403] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.347s user 0.005s sys 0.004s
I20260926 01:55:55.060374 9341 raft_consensus.cc:359] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.061174 9341 raft_consensus.cc:385] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:55.061381 9339 tablet_bootstrap.cc:654] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:55.061482 9341 raft_consensus.cc:740] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:55.062258 9341 consensus_queue.cc:260] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.065629 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 5cbf646569e6497dac90e308f0681372 (DEFAULT_TABLE table=default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:54.976204 9875 tablet.cc:1751] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:55.066741 9194 tablet_service.cc:1558] Processing DeleteTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 6752df6a58914df5b8718dd0439aa885) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:55.067390 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 0d068e93fd374a6091d7473db069a7fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:55.074796 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbf646569e6497dac90e308f0681372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.087232 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 5eef6a48e87e4c61a23058942a82bccc (DEFAULT_TABLE table=default.Table_b5bbdf5281dc45739e2ee7bf9aa6a02c [id=37400e3a991842198609512f0a9aeead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.088340 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eef6a48e87e4c61a23058942a82bccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.091044 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 0c6cccff3de64f3384ce58bb84c43487 (DEFAULT_TABLE table=default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.092516 9049 tablet_service.cc:1511] Processing CreateTablet for tablet b778c95327fc477ca210cb3c983665c1 (DEFAULT_TABLE table=default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.094569 9711 ts_tablet_manager.cc:1916] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:55.100981 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 59362.3us
I20260926 01:55:55.103436 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6cccff3de64f3384ce58bb84c43487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.104943 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b778c95327fc477ca210cb3c983665c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.111461 9194 tablet_service.cc:1511] Processing CreateTablet for tablet 3f1d71caf78f4c75b144af14dce19d70 (DEFAULT_TABLE table=default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.112766 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1d71caf78f4c75b144af14dce19d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.126935 9829 raft_consensus.cc:493] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:55.128634 9829 raft_consensus.cc:515] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.130738 9829 leader_election.cc:290] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:55.132481 9046 tablet_service.cc:1511] Processing CreateTablet for tablet bcb000f9234d4382ad019f63015b92e7 (DEFAULT_TABLE table=default.Table_d1175fab270f48b19ec70a1957bd6af6 [id=88aedd8765374c6395d82b3752c327ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.133529 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcb000f9234d4382ad019f63015b92e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.149582 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c6cccff3de64f3384ce58bb84c43487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.159410 9047 tablet_service.cc:1511] Processing CreateTablet for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 (DEFAULT_TABLE table=default.Table_5bb843a35ff742499a87f56bb5d77d32 [id=760b382db3324c0a870435e2e4083241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.162945 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.167241 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.158933 9340 consensus_queue.cc:260] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.168403 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 with delete_type TABLET_DATA_DELETED (replaced by tablet b612e8955d5f42dd8658320031fe36ca) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.169303 9808 consensus_queue.cc:1048] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:55.170737 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.174633 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.183548 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:55.051139 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:55.186887 9069 raft_consensus.cc:3060] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.191781 8903 raft_consensus.cc:3060] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.193941 8907 raft_consensus.cc:3060] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:55.195477 9152 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:55.202724 9341 ts_tablet_manager.cc:1434] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260926 01:55:55.203959 9711 ts_tablet_manager.cc:1929] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:55.204311 9341 tablet_bootstrap.cc:492] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:55.204376 9711 log.cc:1199] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/f3bb1643da054583b97efd2e1a9e9c2f
I20260926 01:55:55.205080 9711 ts_tablet_manager.cc:1950] T f3bb1643da054583b97efd2e1a9e9c2f P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:55.195098 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:55.219338 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 157) took 2069 ms. Trace:
I20260926 01:55:55.219731 9711 rpcz_store.cc:276] 0926 01:55:53.150037 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.150152 (+ 115us) service_pool.cc:224] Handling call
0926 01:55:55.015550 (+1865398us) ts_tablet_manager.cc:1151] Deleting tablet f3bb1643da054583b97efd2e1a9e9c2f
0926 01:55:55.015599 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.144750 (+129151us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.203815 (+ 59065us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.219253 (+ 15438us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1789641}
I20260926 01:55:55.221045 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 152) took 2083 ms. Trace:
I20260926 01:55:55.221349 9711 rpcz_store.cc:276] 0926 01:55:53.137391 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.255185 (+117794us) service_pool.cc:224] Handling call
0926 01:55:55.220760 (+1965575us) ts_tablet_manager.cc:1151] Deleting tablet 01052a5e22a74a19919bae9dd0296291
0926 01:55:55.220808 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.220979 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964833}
I20260926 01:55:55.222275 9711 tablet_replica.cc:333] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:55.230366 8898 raft_consensus.cc:3060] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.227919 8904 raft_consensus.cc:3060] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.236375 9711 raft_consensus.cc:2243] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:55.236955 9711 raft_consensus.cc:2272] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:55.257738 9854 raft_consensus.cc:493] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:55.258303 9854 raft_consensus.cc:515] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.260674 9854 leader_election.cc:290] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:55.262056 9003 leader_election.cc:343] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:55.271253 8843 consensus_peers.cc:597] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:55.272100 8843 leader_election.cc:304] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:55.285861 9341 tablet_bootstrap.cc:654] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:55.292054 9812 ts_tablet_manager.cc:1929] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:55.292469 9812 log.cc:1199] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/f3bb1643da054583b97efd2e1a9e9c2f
I20260926 01:55:55.293181 9812 ts_tablet_manager.cc:1950] T f3bb1643da054583b97efd2e1a9e9c2f P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:55.294037 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 169) took 1909 ms. Trace:
I20260926 01:55:55.294673 9793 raft_consensus.cc:493] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:55.295110 9793 raft_consensus.cc:515] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.296983 9793 leader_election.cc:290] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:55.300567 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:55.302544 9843 raft_consensus.cc:2749] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:55.305244 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 3f1d71caf78f4c75b144af14dce19d70 (DEFAULT_TABLE table=default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.306336 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1d71caf78f4c75b144af14dce19d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.310873 9793 raft_consensus.cc:493] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:55.311466 9793 raft_consensus.cc:515] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.316005 9808 consensus_queue.cc:1048] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:55.318150 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:55.323431 9793 leader_election.cc:290] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:55.324292 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:55.351047 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.354749 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 5cbf646569e6497dac90e308f0681372 (DEFAULT_TABLE table=default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:43.764076 9349 server_base.cc:1061] running on GCE node
W20260926 01:55:43.811424 9439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:55:43.716266 9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:55:55.365856 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:55:55.370855 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 5da0c41e2e19409d938f8f8ec29cb91a (DEFAULT_TABLE table=default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.372023 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da0c41e2e19409d938f8f8ec29cb91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.372982 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 5da0c41e2e19409d938f8f8ec29cb91a (DEFAULT_TABLE table=default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.374009 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da0c41e2e19409d938f8f8ec29cb91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.375604 9339 tablet_bootstrap.cc:492] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:55.376111 9339 ts_tablet_manager.cc:1403] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.385s user 0.012s sys 0.001s
I20260926 01:55:55.381225 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet 0d068e93fd374a6091d7473db069a7fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.381835 9047 tablet_service.cc:1558] Processing DeleteTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 6752df6a58914df5b8718dd0439aa885) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:55.382488 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 5eef6a48e87e4c61a23058942a82bccc (DEFAULT_TABLE table=default.Table_b5bbdf5281dc45739e2ee7bf9aa6a02c [id=37400e3a991842198609512f0a9aeead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.383070 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 3f1d71caf78f4c75b144af14dce19d70 (DEFAULT_TABLE table=default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.384984 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 5da0c41e2e19409d938f8f8ec29cb91a (DEFAULT_TABLE table=default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.394657 8536 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_ee6cc37e9698474a8a3194f060b8c028" } modify_external_catalogs: true
I20260926 01:55:55.395227 8536 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_ee6cc37e9698474a8a3194f060b8c028" } modify_external_catalogs: true
I20260926 01:55:55.395499 9340 ts_tablet_manager.cc:1434] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.433s user 0.003s sys 0.006s
I20260926 01:55:55.396979 9340 tablet_bootstrap.cc:492] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:55.397560 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:55:55.405172 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 9bac55da5b164c0f981914d39c87c33e
I20260926 01:55:55.406549 9843 raft_consensus.cc:493] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:55:55.406744 8532 catalog_manager.cc:6934] ReplaceTablet: tablet 9bac55da5b164c0f981914d39c87c33e deleted and replaced by tablet 0558d37a3b5f4bcea43a8da7b99ded09
I20260926 01:55:55.407063 9843 raft_consensus.cc:515] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.409066 8540 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:55.411082 9843 leader_election.cc:290] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:55.414837 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:55.418259 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 3b6daaf274894989b90da364f5da5250 (table default.Table_efe5a3dc3242481484fedd388b7be098 [id=be02dac770c84af69e2ff172417a074a]) successfully deleted
I20260926 01:55:55.418733 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bac55da5b164c0f981914d39c87c33e with delete_type TABLET_DATA_DELETED (replaced by tablet 0558d37a3b5f4bcea43a8da7b99ded09) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:55.419448 9195 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bac55da5b164c0f981914d39c87c33e with delete_type TABLET_DATA_DELETED (replaced by tablet 0558d37a3b5f4bcea43a8da7b99ded09) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:55.294361 9812 rpcz_store.cc:276] 0926 01:55:53.387105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.387414 (+ 309us) service_pool.cc:224] Handling call
0926 01:55:54.825074 (+1437660us) ts_tablet_manager.cc:1151] Deleting tablet f3bb1643da054583b97efd2e1a9e9c2f
0926 01:55:54.825131 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:54.977186 (+152055us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.291901 (+314715us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.293969 (+ 2068us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4032,"tablet-delete.queue_time_us":1437029}
I20260926 01:55:55.424677 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet f3bb1643da054583b97efd2e1a9e9c2f (table default.Table_48399c44f8254aaeaa623db43553e1ca [id=eb2d6f7b9c9a48c49b34d0617267301e]) successfully deleted
I20260926 01:55:55.421854 9339 raft_consensus.cc:359] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.427018 9339 raft_consensus.cc:385] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:55.427115 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:55.427321 9339 raft_consensus.cc:740] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:55.427975 9339 consensus_queue.cc:260] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.429524 9339 ts_tablet_manager.cc:1434] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.053s user 0.004s sys 0.000s
I20260926 01:55:55.429862 8535 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_6f6a2a6f414644f7b5d7a322965922fa" } new_table_name: "default.Table_3e325156f7684b6e991c8cb0ed626c25" modify_external_catalogs: true
I20260926 01:55:55.434569 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 170) took 2046 ms. Trace:
W20260926 01:55:55.437979 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9f54f648a55404fb6622b35d84368ee consensus state has no leader
I20260926 01:55:55.440141 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet f3bb1643da054583b97efd2e1a9e9c2f (table default.Table_48399c44f8254aaeaa623db43553e1ca [id=eb2d6f7b9c9a48c49b34d0617267301e]) successfully deleted
W20260926 01:55:55.441424 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 01052a5e22a74a19919bae9dd0296291 because the tablet was not found. No further retry: Not found: Tablet not found: 01052a5e22a74a19919bae9dd0296291
I20260926 01:55:55.441680 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 6f7419abcc2c482e97df70f03e047450 (DEFAULT_TABLE table=default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.442927 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7419abcc2c482e97df70f03e047450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.448855 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:55.434965 9812 rpcz_store.cc:276] 0926 01:55:53.388386 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.389472 (+ 1086us) service_pool.cc:224] Handling call
0926 01:55:55.434193 (+2044721us) ts_tablet_manager.cc:1151] Deleting tablet dccbfbc9fb5947f9b85ca8560e205b08
0926 01:55:55.434248 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.434489 (+ 241us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2044137}
I20260926 01:55:55.451215 9812 tablet_replica.cc:333] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
W20260926 01:55:55.451509 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet dccbfbc9fb5947f9b85ca8560e205b08 because the tablet was not found. No further retry: Not found: Tablet not found: dccbfbc9fb5947f9b85ca8560e205b08
I20260926 01:55:55.452108 9812 raft_consensus.cc:2243] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:55.453085 9812 raft_consensus.cc:2272] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:55.455134 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:55.455981 9349 hybrid_clock.cc:648] HybridClock initialized: now 1790387755455842 us; error 122 us; skew 500 ppm
W20260926 01:55:55.456135 9006 leader_election.cc:343] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:55.456013 9854 consensus_queue.cc:1048] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:55:55.456526 9006 leader_election.cc:304] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:55.457247 9349 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:55:55.472245 8883 tablet_service.cc:1511] Processing CreateTablet for tablet e55a31bb9b9d4862b6bcef71df0cdaea (DEFAULT_TABLE table=default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.472565 9340 tablet_bootstrap.cc:654] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:55.473361 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55a31bb9b9d4862b6bcef71df0cdaea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.483837 9891 raft_consensus.cc:493] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:55:55.484400 9891 raft_consensus.cc:515] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.491854 9891 leader_election.cc:290] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:55.496690 9812 ts_tablet_manager.cc:1916] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:55.498713 9711 ts_tablet_manager.cc:1916] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:55.499404 9196 tablet_service.cc:1511] Processing CreateTablet for tablet e55a31bb9b9d4862b6bcef71df0cdaea (DEFAULT_TABLE table=default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.509550 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
I20260926 01:55:55.511016 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:55:55.517705 9710 ts_tablet_manager.cc:1929] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:55.523427 9349 webserver.cc:533] Webserver started at http://127.8.76.188:33837/ using document root <none> and password file <none>
I20260926 01:55:55.524974 9349 fs_manager.cc:362] Metadata directory not provided
I20260926 01:55:55.525316 9349 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:55:55.530635 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 3c26f557ac274973824e6f9a4532de74 (DEFAULT_TABLE table=default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.531760 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c26f557ac274973824e6f9a4532de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.534137 8879 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab082379974486bbea9f87f5aa64e10 (DEFAULT_TABLE table=default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.535315 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab082379974486bbea9f87f5aa64e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.546775 9340 tablet_bootstrap.cc:492] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:55.547263 9340 ts_tablet_manager.cc:1403] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.150s user 0.012s sys 0.000s
I20260926 01:55:55.550674 9340 raft_consensus.cc:359] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.551337 9340 raft_consensus.cc:385] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:55.511312 8533 master_service.cc:960] ReplaceTablet: received request to replace tablet 1743e863cdd84f3f8ca413fa81cd1914 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:55.568425 8538 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_ee6cc37e9698474a8a3194f060b8c028 [id=0dae284524224264b7ccbd59cd975030] alter complete (version 1)
I20260926 01:55:55.569965 8881 tablet_service.cc:1511] Processing CreateTablet for tablet f74618f7c46744d2a08cb7b4e389c9f3 (DEFAULT_TABLE table=default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.571130 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f74618f7c46744d2a08cb7b4e389c9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.571133 9349 fs_manager.cc:714] Time spent opening directory manager: real 0.035s user 0.007s sys 0.005s
I20260926 01:55:55.551605 9340 raft_consensus.cc:740] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:55.581547 9340 consensus_queue.cc:260] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.587091 8902 raft_consensus.cc:3060] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.592583 9907 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:55:55.594476 9349 fs_manager.cc:730] Time spent opening block manager: real 0.020s user 0.004s sys 0.004s
I20260926 01:55:55.595023 9349 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
uuid: "f75ba46482de422ea23daf17aa100849"
format_stamp: "Formatted at 2026-09-26 01:55:29 on dist-test-slave-5pk1"
I20260926 01:55:55.601011 9349 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:55:55.627362 8907 raft_consensus.cc:3060] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.631253 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_83505dff1c1149ebb8e82a18792b4fa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:55.632939 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83505dff1c1149ebb8e82a18792b4fa3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:55.519701 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 6f7419abcc2c482e97df70f03e047450 (DEFAULT_TABLE table=default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.545550 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab082379974486bbea9f87f5aa64e10 (DEFAULT_TABLE table=default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.549129 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 3c26f557ac274973824e6f9a4532de74 (DEFAULT_TABLE table=default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.580134 9193 tablet_service.cc:1511] Processing CreateTablet for tablet f74618f7c46744d2a08cb7b4e389c9f3 (DEFAULT_TABLE table=default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.597990 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:55.598569 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55a31bb9b9d4862b6bcef71df0cdaea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.606310 9858 consensus_queue.cc:1048] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:55.664894 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f74618f7c46744d2a08cb7b4e389c9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.665511 9808 consensus_queue.cc:1048] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:55.670288 9340 ts_tablet_manager.cc:1434] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.123s user 0.006s sys 0.000s
I20260926 01:55:55.670717 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab082379974486bbea9f87f5aa64e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.619894 9341 tablet_bootstrap.cc:492] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:55.672734 9341 ts_tablet_manager.cc:1403] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.469s user 0.007s sys 0.005s
I20260926 01:55:55.676553 9194 tablet_service.cc:1511] Processing CreateTablet for tablet 030f4d683e404fbd8d2e6863e18f71c0 (DEFAULT_TABLE table=default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.677781 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030f4d683e404fbd8d2e6863e18f71c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.681627 9341 raft_consensus.cc:359] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.682346 9341 raft_consensus.cc:385] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:55.682641 9341 raft_consensus.cc:740] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:55.682991 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7419abcc2c482e97df70f03e047450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.683326 9341 consensus_queue.cc:260] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.685422 9341 ts_tablet_manager.cc:1434] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20260926 01:55:55.688597 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c26f557ac274973824e6f9a4532de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.699693 9808 raft_consensus.cc:493] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:55.700243 9808 raft_consensus.cc:515] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.703050 9808 leader_election.cc:290] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:55.704398 8883 tablet_service.cc:1511] Processing CreateTablet for tablet 030f4d683e404fbd8d2e6863e18f71c0 (DEFAULT_TABLE table=default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:55.705593 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030f4d683e404fbd8d2e6863e18f71c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.710664 9710 log.cc:1199] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/03086ec5451743fdbf2d5726d43f31b1
I20260926 01:55:55.711650 9710 ts_tablet_manager.cc:1950] T 03086ec5451743fdbf2d5726d43f31b1 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:55.713804 8902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:55.715335 9152 leader_election.cc:343] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:55.719899 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_b5bbdf5281dc45739e2ee7bf9aa6a02c" } new_table_name: "default.Table_4a4b59ee98dd4139b83d181413443621" modify_external_catalogs: true
I20260926 01:55:55.727417 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_23c61cea9e0648838bc80efc34b50799"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:55.728982 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23c61cea9e0648838bc80efc34b50799 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:55.729542 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet 6d56f3a5dfed4ac78f1da92e22db558a from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:55.731653 9812 ts_tablet_manager.cc:1929] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:55.738798 8534 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 16f49a5a10ea4f7a821d89f18e3af560
I20260926 01:55:55.743822 8534 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 227) took 1158 ms. Trace:
I20260926 01:55:55.747949 9199 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f49a5a10ea4f7a821d89f18e3af560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:55.744179 8534 rpcz_store.cc:276] 0926 01:55:54.585380 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.586310 (+ 930us) service_pool.cc:224] Handling call
0926 01:55:54.587886 (+ 1576us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:54.805700 (+217814us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:54.805860 (+ 160us) catalog_manager.cc:2863] Locking tablets
0926 01:55:55.087407 (+281547us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:55.738386 (+650979us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:55.738624 (+ 238us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:55.738712 (+ 88us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:55.743737 (+ 5025us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:55.629988 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:55.630413 (+ 425us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:55.630438 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:55:55.630472 (+ 34us) tablet.cc:662] Decoding operations
0926 01:55:55.631987 (+ 1515us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:55.632029 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:55.632053 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:55.632341 (+ 288us) tablet.cc:701] Row locks acquired
0926 01:55:55.632356 (+ 15us) write_op.cc:260] PREPARE: finished
0926 01:55:55.632470 (+ 114us) write_op.cc:270] Start()
0926 01:55:55.632562 (+ 92us) write_op.cc:276] Timestamp: P: 1790387755632418 usec, L: 0
0926 01:55:55.632578 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:55.633088 (+ 510us) log.cc:844] Serialized 728 byte log entry
0926 01:55:55.735857 (+102769us) op_driver.cc:464] REPLICATION: finished
0926 01:55:55.737012 (+ 1155us) write_op.cc:301] APPLY: starting
0926 01:55:55.737077 (+ 65us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:55.737203 (+ 126us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:55.737304 (+ 101us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:55.737551 (+ 247us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:55.737690 (+ 139us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:55.737731 (+ 41us) write_op.cc:312] APPLY: finished
0926 01:55:55.737918 (+ 187us) log.cc:844] Serialized 29 byte log entry
0926 01:55:55.738089 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:55.738171 (+ 82us) write_op.cc:454] Released schema lock
0926 01:55:55.738204 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:55.738432 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:55.738495 (+ 63us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":997,"apply.run_cpu_time_us":1392,"apply.run_wall_time_us":1392,"num_ops":2,"prepare.queue_time_us":540716,"prepare.run_cpu_time_us":2318,"prepare.run_wall_time_us":86747,"raft.queue_time_us":76,"raft.run_cpu_time_us":759,"raft.run_wall_time_us":1727,"replication_time_us":103182}]]}
W20260926 01:55:55.753535 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 681900e0fac74cdabaea1d8a7fb4a189 consensus state has no leader
I20260926 01:55:55.754088 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f49a5a10ea4f7a821d89f18e3af560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:55.754833 9340 tablet_bootstrap.cc:492] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:55.772370 9341 tablet_bootstrap.cc:492] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:55.773141 9909 log.cc:927] Time spent T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260926 01:55:55.776185 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 177) took 2470 ms. Trace:
I20260926 01:55:55.776556 9710 rpcz_store.cc:276] 0926 01:55:53.305383 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.305550 (+ 167us) service_pool.cc:224] Handling call
0926 01:55:54.772951 (+1467401us) ts_tablet_manager.cc:1151] Deleting tablet 03086ec5451743fdbf2d5726d43f31b1
0926 01:55:54.772993 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.130935 (+357942us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.517553 (+386618us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.776103 (+258550us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339360}
I20260926 01:55:55.778380 9710 tablet_replica.cc:333] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:55.779232 9710 raft_consensus.cc:2243] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:55.779704 9710 raft_consensus.cc:2272] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:55.781587 9710 ts_tablet_manager.cc:1916] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:55.783516 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]) successfully deleted
W20260926 01:55:55.796198 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 338410us
I20260926 01:55:55.798012 9006 leader_election.cc:304] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:55.808113 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbf646569e6497dac90e308f0681372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.810972 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c27f1dd7bb0449a6a3fa371a8636e09b" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:55.813303 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee418347ebb04927913810d539ef57ea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:55.814828 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39cd9130640d4b14beef42e9f1b36b99" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:55.816210 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bf2fa5fbe4146b5adb27ff09de878d3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:55.817167 9073 raft_consensus.cc:2393] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
W20260926 01:55:55.819976 9069 consensus_meta.cc:287] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.632s user 0.007s sys 0.001s
I20260926 01:55:55.820464 9069 raft_consensus.cc:2468] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:55.827041 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da0c41e2e19409d938f8f8ec29cb91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.830005 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eef6a48e87e4c61a23058942a82bccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.831341 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1d71caf78f4c75b144af14dce19d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:55.839860 9854 raft_consensus.cc:2749] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:55.840314 9339 tablet_bootstrap.cc:492] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:55.843532 9703 raft_consensus.cc:2749] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:55.843950 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_49bdcbf4ba8b4771b549ee062065c8e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:55.844194 9703 consensus_queue.cc:1048] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:55:55.846056 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49bdcbf4ba8b4771b549ee062065c8e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:55.732098 9812 log.cc:1199] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/03086ec5451743fdbf2d5726d43f31b1
I20260926 01:55:55.847612 9812 ts_tablet_manager.cc:1950] T 03086ec5451743fdbf2d5726d43f31b1 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
W20260926 01:55:55.853288 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 57626.6us
W20260926 01:55:55.854835 9003 leader_election.cc:343] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:55.858246 9339 tablet_bootstrap.cc:654] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:55.860383 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 62704.1us
W20260926 01:55:55.863523 8535 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 229f4cdd841949e4a12e8555cdfe42cc consensus state has no leader
I20260926 01:55:55.868782 9340 tablet_bootstrap.cc:654] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:55.876477 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 172) took 2441 ms. Trace:
I20260926 01:55:55.877039 9812 rpcz_store.cc:276] 0926 01:55:53.435452 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.436584 (+ 1132us) service_pool.cc:224] Handling call
0926 01:55:55.451041 (+2014457us) ts_tablet_manager.cc:1151] Deleting tablet 03086ec5451743fdbf2d5726d43f31b1
0926 01:55:55.451090 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.554805 (+103715us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.731462 (+176657us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:55.876384 (+144922us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882428}
I20260926 01:55:55.878744 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]) successfully deleted
I20260926 01:55:55.879145 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 171) took 2488 ms. Trace:
I20260926 01:55:55.879458 9812 rpcz_store.cc:276] 0926 01:55:53.390432 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.434662 (+ 44230us) service_pool.cc:224] Handling call
0926 01:55:55.878813 (+2444151us) ts_tablet_manager.cc:1151] Deleting tablet 01052a5e22a74a19919bae9dd0296291
0926 01:55:55.878863 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.879072 (+ 209us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310035}
I20260926 01:55:55.822391 9341 tablet_bootstrap.cc:654] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:55.894862 9006 leader_election.cc:304] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:55.898073 9829 raft_consensus.cc:2804] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:55.898681 9829 raft_consensus.cc:697] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:55:55.899782 9829 consensus_queue.cc:237] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:55.902712 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 74322.4us
W20260926 01:55:55.905723 9006 consensus_peers.cc:597] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75d4598596ff49378fc3536fff0ac1b1. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:55.906731 9914 consensus_queue.cc:1048] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:55:55.928002 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 01052a5e22a74a19919bae9dd0296291 because the tablet was not found. No further retry: Not found: Tablet not found: 01052a5e22a74a19919bae9dd0296291
I20260926 01:55:55.928363 9812 tablet_replica.cc:333] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:55.929235 9812 raft_consensus.cc:2243] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:55.929325 9341 tablet_bootstrap.cc:492] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:55.929780 9341 ts_tablet_manager.cc:1403] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.158s user 0.010s sys 0.000s
I20260926 01:55:55.934489 9812 raft_consensus.cc:2272] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:55.933400 9340 tablet_bootstrap.cc:492] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:55.937160 9340 ts_tablet_manager.cc:1403] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.183s user 0.010s sys 0.002s
I20260926 01:55:55.941394 9341 raft_consensus.cc:359] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.942073 9341 raft_consensus.cc:385] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:55.945115 9340 raft_consensus.cc:359] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:55.939422 9812 ts_tablet_manager.cc:1916] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:55.955495 8904 raft_consensus.cc:3060] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:55.942358 9341 raft_consensus.cc:740] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:55.960378 9341 consensus_queue.cc:260] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.963207 9349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:55:55.965435 9349 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:55:55.966043 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:55:55.969848 9917 raft_consensus.cc:493] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:55.970486 9917 raft_consensus.cc:515] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:55.973222 9917 leader_election.cc:290] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:55.974705 8898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bac55da5b164c0f981914d39c87c33e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:55.976364 9152 leader_election.cc:343] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:56.000517 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_79241696f6954afba998044c7a981755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:56.001024 9340 raft_consensus.cc:385] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.001343 9340 raft_consensus.cc:740] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.002082 9340 consensus_queue.cc:260] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:55:56.002600 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79241696f6954afba998044c7a981755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:56.004647 9710 ts_tablet_manager.cc:1929] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:56.005105 9710 log.cc:1199] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/c36ea09cf15647c0a344c9ba9b289d61
I20260926 01:55:56.005841 9710 ts_tablet_manager.cc:1950] T c36ea09cf15647c0a344c9ba9b289d61 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:56.007472 9340 ts_tablet_manager.cc:1434] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20260926 01:55:55.978009 9341 ts_tablet_manager.cc:1434] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
W20260926 01:55:55.910140 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 53386.8us
I20260926 01:55:56.019822 9341 tablet_bootstrap.cc:492] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:56.024317 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 178) took 2684 ms. Trace:
I20260926 01:55:56.024715 9710 rpcz_store.cc:276] 0926 01:55:53.340074 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.340194 (+ 120us) service_pool.cc:224] Handling call
0926 01:55:55.778197 (+2438003us) ts_tablet_manager.cc:1151] Deleting tablet c36ea09cf15647c0a344c9ba9b289d61
0926 01:55:55.778244 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:55.937457 (+159213us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.004486 (+ 67029us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.024228 (+ 19742us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2344385}
I20260926 01:55:56.026320 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet c36ea09cf15647c0a344c9ba9b289d61 (table default.Table_722ce830650d49ffa0b00028e056e1db [id=b10f8555970c4add9293d27073a1175b]) successfully deleted
I20260926 01:55:56.015317 9703 consensus_queue.cc:1048] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260926 01:55:56.047000 9710 tablet_replica.cc:333] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.047899 9710 raft_consensus.cc:2243] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 01:55:56.031872 9006 leader_election.cc:343] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:56.049904 9006 leader_election.cc:304] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:56.048472 9710 raft_consensus.cc:2272] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.053241 9710 ts_tablet_manager.cc:1916] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.054512 9341 tablet_bootstrap.cc:654] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:56.060480 9006 leader_election.cc:343] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:56.060958 9006 leader_election.cc:304] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:56.063068 9915 raft_consensus.cc:2749] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:56.079300 9341 tablet_bootstrap.cc:492] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:56.079813 9341 ts_tablet_manager.cc:1403] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.000s
I20260926 01:55:56.089632 9341 raft_consensus.cc:359] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.090416 9341 raft_consensus.cc:385] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.090713 9341 raft_consensus.cc:740] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.093928 9884 raft_consensus.cc:2749] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:56.102684 9341 consensus_queue.cc:260] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.103339 9340 tablet_bootstrap.cc:492] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:56.104992 9341 ts_tablet_manager.cc:1434] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260926 01:55:56.106052 9341 tablet_bootstrap.cc:492] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:56.113026 9711 ts_tablet_manager.cc:1929] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:56.114056 9711 log.cc:1199] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/03086ec5451743fdbf2d5726d43f31b1
W20260926 01:55:56.145720 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 106608us
I20260926 01:55:56.146486 8843 leader_election.cc:304] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:56.147183 9891 raft_consensus.cc:2749] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:55:56.134646 9341 tablet_bootstrap.cc:654] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:56.148270 8843 leader_election.cc:304] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:56.137480 9710 ts_tablet_manager.cc:1929] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.149490 9710 log.cc:1199] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/1cba678b66744dcfb4d13ae07ebec33f
I20260926 01:55:56.150264 9710 ts_tablet_manager.cc:1950] T 1cba678b66744dcfb4d13ae07ebec33f P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:56.152530 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 179) took 2811 ms. Trace:
I20260926 01:55:56.152860 9710 rpcz_store.cc:276] 0926 01:55:53.341465 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.341576 (+ 111us) service_pool.cc:224] Handling call
0926 01:55:56.046792 (+2705216us) ts_tablet_manager.cc:1151] Deleting tablet 1cba678b66744dcfb4d13ae07ebec33f
0926 01:55:56.046850 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.111396 (+ 64546us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.137319 (+ 25923us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.152464 (+ 15145us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2612902}
I20260926 01:55:56.154546 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 1cba678b66744dcfb4d13ae07ebec33f (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]) successfully deleted
I20260926 01:55:56.170950 9710 tablet_replica.cc:333] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.171485 9891 raft_consensus.cc:2749] T ee418347ebb04927913810d539ef57ea P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:56.172868 8843 leader_election.cc:304] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:56.173974 8843 leader_election.cc:304] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:56.176735 9341 tablet_bootstrap.cc:492] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:56.177372 9341 ts_tablet_manager.cc:1403] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.003s
I20260926 01:55:56.180260 9341 raft_consensus.cc:359] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.185000 9341 raft_consensus.cc:385] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.185360 9341 raft_consensus.cc:740] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.174115 9710 raft_consensus.cc:2243] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:56.186908 9710 raft_consensus.cc:2272] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.189016 9710 ts_tablet_manager.cc:1916] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.189849 9891 raft_consensus.cc:2749] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:56.191217 9930 raft_consensus.cc:2749] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:55:56.190848 9341 consensus_queue.cc:260] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.133860 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet 3b6daaf274894989b90da364f5da5250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:47 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:56.135098 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 6f7419abcc2c482e97df70f03e047450 (DEFAULT_TABLE table=default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.135499 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 3c26f557ac274973824e6f9a4532de74 (DEFAULT_TABLE table=default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.136227 9051 tablet_service.cc:1511] Processing CreateTablet for tablet e55a31bb9b9d4862b6bcef71df0cdaea (DEFAULT_TABLE table=default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.136368 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab082379974486bbea9f87f5aa64e10 (DEFAULT_TABLE table=default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.137579 9052 tablet_service.cc:1511] Processing CreateTablet for tablet f74618f7c46744d2a08cb7b4e389c9f3 (DEFAULT_TABLE table=default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.139168 9045 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f49a5a10ea4f7a821d89f18e3af560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:56.139278 9048 tablet_service.cc:1511] Processing CreateTablet for tablet 030f4d683e404fbd8d2e6863e18f71c0 (DEFAULT_TABLE table=default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.208295 9341 ts_tablet_manager.cc:1434] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260926 01:55:56.209760 9341 tablet_bootstrap.cc:492] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:56.229930 9341 tablet_bootstrap.cc:654] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:56.236802 9943 tablet.cc:1751] T e53c9eb8a6d648b2a9359f02b7f8764b P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:56.249577 9945 tablet.cc:1751] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:56.265921 9341 tablet_bootstrap.cc:492] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:56.266410 9341 ts_tablet_manager.cc:1403] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.000s
I20260926 01:55:56.269541 9341 raft_consensus.cc:359] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.270246 9341 raft_consensus.cc:385] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.126350 9340 tablet_bootstrap.cc:654] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:56.287622 9341 raft_consensus.cc:740] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.277861 9710 ts_tablet_manager.cc:1929] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.288535 9710 log.cc:1199] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/39cd9130640d4b14beef42e9f1b36b99
I20260926 01:55:56.289376 9710 ts_tablet_manager.cc:1950] T 39cd9130640d4b14beef42e9f1b36b99 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:56.291646 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 180) took 2933 ms. Trace:
I20260926 01:55:56.292471 9341 consensus_queue.cc:260] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.293534 9710 rpcz_store.cc:276] 0926 01:55:53.358516 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.358652 (+ 136us) service_pool.cc:224] Handling call
0926 01:55:56.170763 (+2812111us) ts_tablet_manager.cc:1151] Deleting tablet 39cd9130640d4b14beef42e9f1b36b99
0926 01:55:56.170815 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.220918 (+ 50103us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.277697 (+ 56779us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.291569 (+ 13872us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2736668}
I20260926 01:55:56.295056 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 39cd9130640d4b14beef42e9f1b36b99 (table default.Table_92899e57b6c14f62b2a999bb2487f558 [id=45e5e00e818749a8beb091f6a62b7bf6]) successfully deleted
I20260926 01:55:56.295498 9812 ts_tablet_manager.cc:1929] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.295883 9812 log.cc:1199] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:55:56.296537 9812 ts_tablet_manager.cc:1950] T 55452e70a7fe48efbafdc7cbb29c6272 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:56.297272 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 176) took 2459 ms. Trace:
I20260926 01:55:56.297923 9710 tablet_replica.cc:333] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.297555 9812 rpcz_store.cc:276] 0926 01:55:53.838421 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.838584 (+ 163us) service_pool.cc:224] Handling call
0926 01:55:55.928184 (+2089600us) ts_tablet_manager.cc:1151] Deleting tablet 55452e70a7fe48efbafdc7cbb29c6272
0926 01:55:55.928232 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.095808 (+167576us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.295341 (+199533us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.297214 (+ 1873us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2088870}
I20260926 01:55:56.303223 9710 raft_consensus.cc:2243] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:56.305203 9710 raft_consensus.cc:2272] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:56.298667 9152 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 71091.1us
I20260926 01:55:56.298830 9341 ts_tablet_manager.cc:1434] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.032s user 0.003s sys 0.004s
I20260926 01:55:56.307430 9710 ts_tablet_manager.cc:1916] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.310315 9341 tablet_bootstrap.cc:492] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:56.327373 9341 tablet_bootstrap.cc:654] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:56.331702 8524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260926 01:55:56.345468 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55a31bb9b9d4862b6bcef71df0cdaea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.161357 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bac55da5b164c0f981914d39c87c33e with delete_type TABLET_DATA_DELETED (replaced by tablet 0558d37a3b5f4bcea43a8da7b99ded09) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:56.183990 9339 tablet_bootstrap.cc:492] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:56.347724 9812 tablet_replica.cc:333] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:56.348541 9812 raft_consensus.cc:2243] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:56.349546 9812 raft_consensus.cc:2272] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.352517 9340 tablet_bootstrap.cc:492] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:56.352941 9340 ts_tablet_manager.cc:1403] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.250s user 0.008s sys 0.003s
I20260926 01:55:56.355592 9340 raft_consensus.cc:359] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.356199 9340 raft_consensus.cc:385] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.356444 9340 raft_consensus.cc:740] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.357172 9340 consensus_queue.cc:260] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.362435 9339 ts_tablet_manager.cc:1403] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.523s user 0.020s sys 0.000s
I20260926 01:55:56.363014 9805 raft_consensus.cc:493] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.363498 9805 raft_consensus.cc:515] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.365476 9805 leader_election.cc:290] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:56.366717 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:56.368211 9152 leader_election.cc:343] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:56.368943 9944 tablet.cc:1751] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:55:56.371989 9812 ts_tablet_manager.cc:1916] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.388391 9710 ts_tablet_manager.cc:1929] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.384510 9340 ts_tablet_manager.cc:1434] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20260926 01:55:56.388882 9710 log.cc:1199] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:55:56.393662 9710 ts_tablet_manager.cc:1950] T 55452e70a7fe48efbafdc7cbb29c6272 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:56.395829 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 181) took 2727 ms. Trace:
I20260926 01:55:56.396152 9710 rpcz_store.cc:276] 0926 01:55:53.668586 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.718563 (+ 49977us) service_pool.cc:224] Handling call
0926 01:55:56.297745 (+2579182us) ts_tablet_manager.cc:1151] Deleting tablet 55452e70a7fe48efbafdc7cbb29c6272
0926 01:55:56.297793 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.333877 (+ 36084us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.388183 (+ 54306us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.395756 (+ 7573us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2578355}
I20260926 01:55:56.397239 9710 tablet_replica.cc:333] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.397708 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 55452e70a7fe48efbafdc7cbb29c6272 (table default.Table_cdcfa57f52a747aaa93bb04410df7061 [id=fccbbbc294bb4548813a1229bd9e2ef8]) successfully deleted
I20260926 01:55:56.398005 9710 raft_consensus.cc:2243] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:56.398418 9340 tablet_bootstrap.cc:492] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:56.398571 9805 raft_consensus.cc:493] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.398985 9805 raft_consensus.cc:515] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.401681 9805 leader_election.cc:290] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:56.402858 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:56.404371 9152 leader_election.cc:343] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:56.415917 9710 raft_consensus.cc:2272] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.418067 9710 ts_tablet_manager.cc:1916] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:56.419865 9006 consensus_peers.cc:597] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c36ea09cf15647c0a344c9ba9b289d61. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:55:56.436921 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 105226us
I20260926 01:55:56.438042 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 55452e70a7fe48efbafdc7cbb29c6272 (table default.Table_cdcfa57f52a747aaa93bb04410df7061 [id=fccbbbc294bb4548813a1229bd9e2ef8]) successfully deleted
I20260926 01:55:56.440542 8533 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
W20260926 01:55:56.441340 8524 consensus_peers.cc:597] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba -> Peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Couldn't send request to peer f75ba46482de422ea23daf17aa100849. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260926 01:55:56.421312 9340 tablet_bootstrap.cc:654] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:56.447870 9854 raft_consensus.cc:493] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.469406 9812 ts_tablet_manager.cc:1929] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.469813 9812 log.cc:1199] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/a601d850375c43679dde7a46cfe71d22
I20260926 01:55:56.476430 9710 ts_tablet_manager.cc:1929] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.476785 9710 log.cc:1199] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/a601d850375c43679dde7a46cfe71d22
I20260926 01:55:56.477551 9854 raft_consensus.cc:515] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.485682 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f74618f7c46744d2a08cb7b4e389c9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.494218 9710 ts_tablet_manager.cc:1950] T a601d850375c43679dde7a46cfe71d22 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:56.495666 9854 leader_election.cc:290] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:56.496484 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 184) took 2564 ms. Trace:
I20260926 01:55:56.496805 9710 rpcz_store.cc:276] 0926 01:55:53.931991 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.932127 (+ 136us) service_pool.cc:224] Handling call
0926 01:55:56.397077 (+2464950us) ts_tablet_manager.cc:1151] Deleting tablet a601d850375c43679dde7a46cfe71d22
0926 01:55:56.397120 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.446980 (+ 49860us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.476294 (+ 29314us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.496413 (+ 20119us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2464359}
I20260926 01:55:56.514621 9155 leader_election.cc:304] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:56.518833 9812 ts_tablet_manager.cc:1950] T a601d850375c43679dde7a46cfe71d22 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:56.521307 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 192) took 2126 ms. Trace:
I20260926 01:55:56.521696 9710 rpcz_store.cc:276] 0926 01:55:54.394803 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.395947 (+ 1144us) service_pool.cc:224] Handling call
0926 01:55:56.520875 (+2124928us) ts_tablet_manager.cc:1151] Deleting tablet 0c6cccff3de64f3384ce58bb84c43487
0926 01:55:56.520923 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.521238 (+ 315us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2101420}
I20260926 01:55:56.523831 9805 raft_consensus.cc:2804] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:56.525153 9805 raft_consensus.cc:697] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:56.527096 9891 raft_consensus.cc:493] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.527590 9891 raft_consensus.cc:515] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.530277 9710 tablet_replica.cc:333] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.531103 9710 raft_consensus.cc:2243] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:56.531785 9891 leader_election.cc:290] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:56.531900 9710 raft_consensus.cc:2272] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.532585 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:56.537447 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 180) took 2338 ms. Trace:
I20260926 01:55:56.537813 9812 rpcz_store.cc:276] 0926 01:55:54.198852 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.198974 (+ 122us) service_pool.cc:224] Handling call
0926 01:55:56.347547 (+2148573us) ts_tablet_manager.cc:1151] Deleting tablet a601d850375c43679dde7a46cfe71d22
0926 01:55:56.347597 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.407372 (+ 59775us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.469263 (+ 61891us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.537365 (+ 68102us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2147911}
I20260926 01:55:56.526060 9805 consensus_queue.cc:237] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.543440 9710 ts_tablet_manager.cc:1916] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.544800 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7419abcc2c482e97df70f03e047450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.553946 9812 tablet_replica.cc:333] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:56.554838 9812 raft_consensus.cc:2243] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:56.555363 9812 raft_consensus.cc:2272] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.557210 9812 ts_tablet_manager.cc:1916] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.558808 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab082379974486bbea9f87f5aa64e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.562227 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c26f557ac274973824e6f9a4532de74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.575060 9048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030f4d683e404fbd8d2e6863e18f71c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.579669 9711 ts_tablet_manager.cc:1950] T 03086ec5451743fdbf2d5726d43f31b1 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:56.585237 9940 tablet.cc:1751] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:56.607565 9891 raft_consensus.cc:493] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.608068 9891 raft_consensus.cc:515] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.610116 9891 leader_election.cc:290] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:56.612418 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:56.613339 9219 raft_consensus.cc:2393] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:56.615705 9710 ts_tablet_manager.cc:1929] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.616101 9710 log.cc:1199] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/70ef5265eb6244aa96c8195aa30cdfb0
I20260926 01:55:56.616803 9710 ts_tablet_manager.cc:1950] T 70ef5265eb6244aa96c8195aa30cdfb0 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:56.621888 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:56.622776 9073 raft_consensus.cc:2468] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:56.632355 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 194) took 2166 ms. Trace:
I20260926 01:55:56.632460 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bac55da5b164c0f981914d39c87c33e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:56.632707 9710 rpcz_store.cc:276] 0926 01:55:54.465939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.466111 (+ 172us) service_pool.cc:224] Handling call
0926 01:55:56.530109 (+2063998us) ts_tablet_manager.cc:1151] Deleting tablet 70ef5265eb6244aa96c8195aa30cdfb0
0926 01:55:56.530155 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.582117 (+ 51962us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.615563 (+ 33446us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.632278 (+ 16715us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2063346}
I20260926 01:55:56.633296 9073 raft_consensus.cc:2468] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:56.636425 9339 raft_consensus.cc:359] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.637193 9339 raft_consensus.cc:385] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.637506 9339 raft_consensus.cc:740] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.638545 9710 tablet_replica.cc:333] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.639304 9710 raft_consensus.cc:2243] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:56.642863 9710 raft_consensus.cc:2272] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.647127 9710 ts_tablet_manager.cc:1916] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.684403 9710 ts_tablet_manager.cc:1929] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:56.684784 9710 log.cc:1199] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/c27f1dd7bb0449a6a3fa371a8636e09b
I20260926 01:55:56.685490 9710 ts_tablet_manager.cc:1950] T c27f1dd7bb0449a6a3fa371a8636e09b P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:55:56.691824 9941 log.cc:927] Time spent T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20260926 01:55:56.694218 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 193) took 2229 ms. Trace:
I20260926 01:55:56.694588 9710 rpcz_store.cc:276] 0926 01:55:54.464845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.466565 (+ 1720us) service_pool.cc:224] Handling call
0926 01:55:56.638346 (+2171781us) ts_tablet_manager.cc:1151] Deleting tablet c27f1dd7bb0449a6a3fa371a8636e09b
0926 01:55:56.638390 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.668861 (+ 30471us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.684266 (+ 15405us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.694149 (+ 9883us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171173}
I20260926 01:55:56.699225 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 195) took 2232 ms. Trace:
I20260926 01:55:56.699537 9710 rpcz_store.cc:276] 0926 01:55:54.466844 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.467667 (+ 823us) service_pool.cc:224] Handling call
0926 01:55:56.698935 (+2231268us) ts_tablet_manager.cc:1151] Deleting tablet 32e0fe60b8744dbb8ae2bafefe3e834d
0926 01:55:56.698979 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.699166 (+ 187us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230786}
I20260926 01:55:56.702584 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 196) took 2235 ms. Trace:
I20260926 01:55:56.702883 9710 rpcz_store.cc:276] 0926 01:55:54.467552 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.467653 (+ 101us) service_pool.cc:224] Handling call
0926 01:55:56.702317 (+2234664us) ts_tablet_manager.cc:1151] Deleting tablet f3bb1643da054583b97efd2e1a9e9c2f
0926 01:55:56.702359 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.702532 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2233695}
I20260926 01:55:56.705348 9341 tablet_bootstrap.cc:492] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:56.705806 9341 ts_tablet_manager.cc:1403] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.396s user 0.014s sys 0.069s
I20260926 01:55:56.706775 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 198) took 1641 ms. Trace:
I20260926 01:55:56.707093 9710 rpcz_store.cc:276] 0926 01:55:55.065204 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.066594 (+ 1390us) service_pool.cc:224] Handling call
0926 01:55:56.706432 (+1639838us) ts_tablet_manager.cc:1151] Deleting tablet 17b8b0d54f5645e5963f1de217f94c84
0926 01:55:56.706490 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.706724 (+ 234us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639270}
I20260926 01:55:56.709975 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 199) took 1644 ms. Trace:
I20260926 01:55:56.709746 9341 raft_consensus.cc:359] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.710496 9341 raft_consensus.cc:385] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.710784 9341 raft_consensus.cc:740] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.710290 9710 rpcz_store.cc:276] 0926 01:55:55.065501 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.067268 (+ 1767us) service_pool.cc:224] Handling call
0926 01:55:56.709663 (+1642395us) ts_tablet_manager.cc:1151] Deleting tablet 0d068e93fd374a6091d7473db069a7fa
0926 01:55:56.709701 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.709927 (+ 226us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1641903}
I20260926 01:55:56.711819 9341 consensus_queue.cc:260] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.713449 9710 tablet_replica.cc:333] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.714193 9710 raft_consensus.cc:2243] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:56.714617 9710 raft_consensus.cc:2272] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.716370 9710 ts_tablet_manager.cc:1916] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.720366 9339 consensus_queue.cc:260] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.730371 9341 ts_tablet_manager.cc:1434] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20260926 01:55:56.740299 9812 ts_tablet_manager.cc:1929] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:56.740748 9812 log.cc:1199] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/1df1d4daaddc4ac4aa02aa7763ea3184
I20260926 01:55:56.741407 9812 ts_tablet_manager.cc:1950] T 1df1d4daaddc4ac4aa02aa7763ea3184 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:56.498178 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet a601d850375c43679dde7a46cfe71d22 (table default.Table_caaf07e0f7d24f7388b358362fd1668f [id=6b7b2329cccd4e96acfc168a00029171]) successfully deleted
W20260926 01:55:56.746881 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 248719us
W20260926 01:55:56.748088 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 0c6cccff3de64f3384ce58bb84c43487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c6cccff3de64f3384ce58bb84c43487 already in progress: creating tablet
I20260926 01:55:56.748729 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 70ef5265eb6244aa96c8195aa30cdfb0 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:55:56.749300 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet c27f1dd7bb0449a6a3fa371a8636e09b (table default.Table_29ce250e9d9a4162966731d01a8e3b23 [id=e664d2aa0cf74dfbad934ad658e8ec0e]) successfully deleted
W20260926 01:55:56.749876 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 32e0fe60b8744dbb8ae2bafefe3e834d because the tablet was not found. No further retry: Not found: Tablet not found: 32e0fe60b8744dbb8ae2bafefe3e834d
W20260926 01:55:56.500118 8537 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5eef6a48e87e4c61a23058942a82bccc consensus state has no leader
I20260926 01:55:56.506815 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 6d56f3a5dfed4ac78f1da92e22db558a
I20260926 01:55:56.750891 8537 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 243) took 1109 ms. Trace:
I20260926 01:55:56.751775 8531 catalog_manager.cc:6934] ReplaceTablet: tablet 6d56f3a5dfed4ac78f1da92e22db558a deleted and replaced by tablet 1c5c66728bb54ec6beb4f0a2c25d9105
I20260926 01:55:56.752122 8531 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 244) took 1055 ms. Trace:
W20260926 01:55:56.750432 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet f3bb1643da054583b97efd2e1a9e9c2f because the tablet was not found. No further retry: Not found: Tablet not found: f3bb1643da054583b97efd2e1a9e9c2f
W20260926 01:55:56.753057 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 17b8b0d54f5645e5963f1de217f94c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17b8b0d54f5645e5963f1de217f94c84 already in progress: creating tablet
W20260926 01:55:56.753675 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 0d068e93fd374a6091d7473db069a7fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d068e93fd374a6091d7473db069a7fa already in progress: creating tablet
I20260926 01:55:56.685937 8532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 242) took 1082 ms. Trace:
I20260926 01:55:56.755236 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51458 (request call id 180) took 1403 ms. Trace:
I20260926 01:55:56.755702 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d56f3a5dfed4ac78f1da92e22db558a with delete_type TABLET_DATA_DELETED (replaced by tablet 1c5c66728bb54ec6beb4f0a2c25d9105) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:56.756404 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 315078us
I20260926 01:55:56.754742 8532 rpcz_store.cc:276] 0926 01:55:55.603687 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.603842 (+ 155us) service_pool.cc:224] Handling call
0926 01:55:55.727321 (+123479us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607566 00007f717c1b2034 00007f717c1b1fc2 00007f717c1b1ef1 00007f717c1b1e6c 00007f717c1b1122 00007f717a45c09c 00007f717a45b4d6
0926 01:55:55.729356 (+ 2035us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:55:55.744792 (+ 15436us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:55:56.685225 (+940433us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:55:56.685532 (+ 307us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:55:56.685836 (+ 304us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:56.304933 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:56.305335 (+ 402us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:56.305358 (+ 23us) write_op.cc:435] Acquired schema lock
0926 01:55:56.305372 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:56.305617 (+ 245us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:56.305654 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:56.305677 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:56.305857 (+ 180us) tablet.cc:701] Row locks acquired
0926 01:55:56.305870 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:56.305966 (+ 96us) write_op.cc:270] Start()
0926 01:55:56.306041 (+ 75us) write_op.cc:276] Timestamp: P: 1790387756305901 usec, L: 0
0926 01:55:56.306055 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:56.359490 (+ 53435us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x7b5400030d20. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c70aa50 00007f717c794302 00007f717c789727 00007f717c789564 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11
0926 01:55:56.360059 (+ 569us) log.cc:844] Serialized 433 byte log entry
0926 01:55:56.683162 (+323103us) op_driver.cc:464] REPLICATION: finished
0926 01:55:56.683670 (+ 508us) write_op.cc:301] APPLY: starting
0926 01:55:56.683745 (+ 75us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:56.683874 (+ 129us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:56.683897 (+ 23us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:56.684368 (+ 471us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:56.684514 (+ 146us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:55:56.684554 (+ 40us) write_op.cc:312] APPLY: finished
0926 01:55:56.684758 (+ 204us) log.cc:844] Serialized 29 byte log entry
0926 01:55:56.684921 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.684990 (+ 69us) write_op.cc:454] Released schema lock
0926 01:55:56.685026 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:56.685268 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.685292 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":282140544,"child_traces":[["op",{"apply.queue_time_us":348,"apply.run_cpu_time_us":1565,"apply.run_wall_time_us":1566,"num_ops":2,"prepare.queue_time_us":558922,"prepare.run_cpu_time_us":3542,"prepare.run_wall_time_us":55378,"replication_time_us":376981,"spinlock_wait_cycles":119232384}]]}
I20260926 01:55:56.758041 8883 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d56f3a5dfed4ac78f1da92e22db558a with delete_type TABLET_DATA_DELETED (replaced by tablet 1c5c66728bb54ec6beb4f0a2c25d9105) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:56.697631 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_df0e68c2c9ea42e18e91228cbf77f457" } modify_external_catalogs: true
W20260926 01:55:56.759079 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8e0e6ede1ce4487b08bfd5743e3656a consensus state has no leader
I20260926 01:55:56.758955 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_df0e68c2c9ea42e18e91228cbf77f457" } modify_external_catalogs: true
I20260926 01:55:56.702576 8535 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_c5aafc7dae5740a9bceb09dc9fa995c7" } new_table_name: "default.Table_a8ab0ff60e564bac9e08a9a1c7736a10" modify_external_catalogs: true
I20260926 01:55:56.760684 9341 tablet_bootstrap.cc:492] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:55:56.761091 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 307632us
I20260926 01:55:56.766866 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet a601d850375c43679dde7a46cfe71d22 (table default.Table_caaf07e0f7d24f7388b358362fd1668f [id=6b7b2329cccd4e96acfc168a00029171]) successfully deleted
I20260926 01:55:56.765560 9340 tablet_bootstrap.cc:492] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:56.767697 9340 ts_tablet_manager.cc:1403] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.369s user 0.024s sys 0.014s
I20260926 01:55:56.774418 9710 ts_tablet_manager.cc:1929] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:56.774899 9710 log.cc:1199] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/9bac55da5b164c0f981914d39c87c33e
I20260926 01:55:56.775741 9710 ts_tablet_manager.cc:1950] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:56.776960 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 203) took 1357 ms. Trace:
I20260926 01:55:56.777248 9710 rpcz_store.cc:276] 0926 01:55:55.419205 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.419310 (+ 105us) service_pool.cc:224] Handling call
0926 01:55:56.713301 (+1293991us) ts_tablet_manager.cc:1151] Deleting tablet 9bac55da5b164c0f981914d39c87c33e
0926 01:55:56.713340 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.735126 (+ 21786us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.774258 (+ 39132us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.776899 (+ 2641us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293464}
I20260926 01:55:56.779552 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 210) took 1032 ms. Trace:
I20260926 01:55:56.779858 9710 rpcz_store.cc:276] 0926 01:55:55.747636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.747779 (+ 143us) service_pool.cc:224] Handling call
0926 01:55:56.779193 (+1031414us) ts_tablet_manager.cc:1151] Deleting tablet 16f49a5a10ea4f7a821d89f18e3af560
0926 01:55:56.779238 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.779501 (+ 263us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030815}
I20260926 01:55:56.790256 9805 raft_consensus.cc:493] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.791100 9805 raft_consensus.cc:515] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:56.755617 9944 rpcz_store.cc:276] 0926 01:55:55.352002 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.352194 (+ 192us) service_pool.cc:224] Handling call
0926 01:55:56.755137 (+1402943us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:55.353467 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:55:55.353872 (+ 405us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:55:55.353904 (+ 32us) alter_schema_op.cc:66] Acquired schema lock
0926 01:55:55.353949 (+ 45us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:55:55.354089 (+ 140us) alter_schema_op.cc:128] START. Timestamp: 1790387755353982
0926 01:55:55.354151 (+ 62us) op_driver.cc:348] REPLICATION: starting
0926 01:55:55.354864 (+ 713us) log.cc:844] Serialized 173 byte log entry
0926 01:55:56.165608 (+810744us) op_driver.cc:464] REPLICATION: finished
0926 01:55:56.368691 (+203083us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:55:56.370410 (+ 1719us) log.cc:844] Serialized 5 byte log entry
0926 01:55:56.683239 (+312829us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.749364 (+ 66125us) log.cc:844] Serialized 14 byte log entry
0926 01:55:56.754940 (+ 5576us) alter_schema_op.cc:73] Released schema lock
0926 01:55:56.754972 (+ 32us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":202824,"prepare.queue_time_us":97,"prepare.run_cpu_time_us":10767,"prepare.run_wall_time_us":221327,"replication_time_us":810856,"thread_start_us":498694,"threads_started":2,"wal-append.queue_time_us":176095}]]}
I20260926 01:55:56.795536 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:56.797077 9152 leader_election.cc:343] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:56.797765 9805 leader_election.cc:290] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:56.806937 9340 raft_consensus.cc:359] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.813670 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:56.814359 9218 raft_consensus.cc:2468] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:55:56.818740 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 309820us
I20260926 01:55:56.859954 8534 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 2e12827ad93c47cf9013e238215ceb7a
W20260926 01:55:56.860026 8524 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 93177.8us
I20260926 01:55:56.860647 9340 raft_consensus.cc:385] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:56.862258 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 158) took 3558 ms. Trace:
I20260926 01:55:56.862759 9711 rpcz_store.cc:276] 0926 01:55:53.304073 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.345253 (+ 41180us) service_pool.cc:224] Handling call
0926 01:55:55.222124 (+1876871us) ts_tablet_manager.cc:1151] Deleting tablet 03086ec5451743fdbf2d5726d43f31b1
0926 01:55:55.222165 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.040848 (+818683us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.112742 (+ 71894us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.862171 (+749429us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20434,"tablet-delete.queue_time_us":1843183}
I20260926 01:55:56.864507 9340 raft_consensus.cc:740] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:56.865281 9340 consensus_queue.cc:260] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.772109 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_4c909c1cd0a54659b097c9ef7f487867"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:56.871909 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c909c1cd0a54659b097c9ef7f487867 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:56.778708 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 9bac55da5b164c0f981914d39c87c33e (table default.Table_3478e66c0ff74bc4a27eb418e346e4e2 [id=05847c8c0e744754b5456a6773f5e41f]) successfully deleted
W20260926 01:55:56.874709 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 96021.8us
I20260926 01:55:56.871315 8540 catalog_manager.cc:5838] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:56.875839 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:56.876515 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]). Expected version 1 got 0
W20260926 01:55:56.877686 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 16f49a5a10ea4f7a821d89f18e3af560 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f49a5a10ea4f7a821d89f18e3af560 already in progress: creating tablet
I20260926 01:55:56.751278 8537 rpcz_store.cc:276] 0926 01:55:55.641299 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.641854 (+ 555us) service_pool.cc:224] Handling call
0926 01:55:55.719793 (+ 77939us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607be6 00007f717c1bb634 00007f717c1bb5c2 00007f717c1bb4f1 00007f717c1bb46c 00007f717c1ba722 00007f717a45c09c 00007f717a45b4d6
0926 01:55:55.720518 (+ 725us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:55.722267 (+ 1749us) catalog_manager.cc:3764] Apply alter schema
0926 01:55:55.742957 (+ 20690us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:55:55.743184 (+ 227us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:55:56.499653 (+756469us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:55:56.750788 (+251135us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:56.302996 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:56.303379 (+ 383us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:56.303401 (+ 22us) write_op.cc:435] Acquired schema lock
0926 01:55:56.303412 (+ 11us) tablet.cc:662] Decoding operations
0926 01:55:56.303666 (+ 254us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:56.303703 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:56.303739 (+ 36us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:56.303881 (+ 142us) tablet.cc:701] Row locks acquired
0926 01:55:56.303892 (+ 11us) write_op.cc:260] PREPARE: finished
0926 01:55:56.304005 (+ 113us) write_op.cc:270] Start()
0926 01:55:56.304089 (+ 84us) write_op.cc:276] Timestamp: P: 1790387756303934 usec, L: 0
0926 01:55:56.304103 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:56.304614 (+ 511us) log.cc:844] Serialized 405 byte log entry
0926 01:55:56.494684 (+190070us) op_driver.cc:464] REPLICATION: finished
0926 01:55:56.497262 (+ 2578us) write_op.cc:301] APPLY: starting
0926 01:55:56.497312 (+ 50us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:56.497400 (+ 88us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:56.497481 (+ 81us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:56.497621 (+ 140us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:56.497740 (+ 119us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:56.497778 (+ 38us) write_op.cc:312] APPLY: finished
0926 01:55:56.497944 (+ 166us) log.cc:844] Serialized 23 byte log entry
0926 01:55:56.498071 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.498121 (+ 50us) write_op.cc:454] Released schema lock
0926 01:55:56.498160 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":177620224,"child_traces":[["op",{"apply.queue_time_us":2454,"apply.run_cpu_time_us":1102,"apply.run_wall_time_us":3172,"num_ops":1,"prepare.queue_time_us":558615,"prepare.run_cpu_time_us":1811,"prepare.run_wall_time_us":1810,"replication_time_us":190485,"thread_start_us":187921,"threads_started":1}]]}
I20260926 01:55:56.752424 8531 rpcz_store.cc:276] 0926 01:55:55.696640 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.729414 (+ 32774us) service_pool.cc:224] Handling call
0926 01:55:56.752063 (+1022649us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:56.300627 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:56.301074 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:56.301101 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:55:56.301113 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:56.301457 (+ 344us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:56.301500 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:56.301525 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:56.301741 (+ 216us) tablet.cc:701] Row locks acquired
0926 01:55:56.301755 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:56.301894 (+ 139us) write_op.cc:270] Start()
0926 01:55:56.301990 (+ 96us) write_op.cc:276] Timestamp: P: 1790387756301815 usec, L: 0
0926 01:55:56.302005 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:56.302634 (+ 629us) log.cc:844] Serialized 625 byte log entry
0926 01:55:56.494384 (+191750us) op_driver.cc:464] REPLICATION: finished
0926 01:55:56.495247 (+ 863us) write_op.cc:301] APPLY: starting
0926 01:55:56.495319 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:56.495448 (+ 129us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:56.495551 (+ 103us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:56.495944 (+ 393us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:56.496178 (+ 234us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:56.496218 (+ 40us) write_op.cc:312] APPLY: finished
0926 01:55:56.496424 (+ 206us) log.cc:844] Serialized 29 byte log entry
0926 01:55:56.496594 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.496673 (+ 79us) write_op.cc:454] Released schema lock
0926 01:55:56.496715 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:56.496962 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.496984 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7208,"child_traces":[["op",{"apply.queue_time_us":682,"apply.run_cpu_time_us":1682,"apply.run_wall_time_us":1681,"num_ops":2,"prepare.queue_time_us":567949,"prepare.run_cpu_time_us":2257,"prepare.run_wall_time_us":2256,"replication_time_us":192257}]]}
I20260926 01:55:56.883139 9197 tablet_service.cc:1511] Processing CreateTablet for tablet e140cf11b8bb49f9a94c0158a2afca4d (DEFAULT_TABLE table=default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.883675 9193 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e12827ad93c47cf9013e238215ceb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:56 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:56.884207 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e140cf11b8bb49f9a94c0158a2afca4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.886639 9339 ts_tablet_manager.cc:1434] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.520s user 0.012s sys 0.000s
W20260926 01:55:56.886960 9339 ts_tablet_manager.cc:1467] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Tablet startup took 520ms
I20260926 01:55:56.887164 9198 tablet_service.cc:1511] Processing CreateTablet for tablet ec749c375d784786ba93020996708bb1 (DEFAULT_TABLE table=default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.888171 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec749c375d784786ba93020996708bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.883028 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 2c0cd3e30f084966b27921e658cacf58 (DEFAULT_TABLE table=default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.892208 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0cd3e30f084966b27921e658cacf58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.879179 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet 3bf2fa5fbe4146b5adb27ff09de878d3 from {username='slave'} at 127.0.0.1:42048
W20260926 01:55:56.878312 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 6d56f3a5dfed4ac78f1da92e22db558a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d56f3a5dfed4ac78f1da92e22db558a already in progress: creating tablet
W20260926 01:55:56.894624 9152 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260926 01:55:56.894825 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 2e12827ad93c47cf9013e238215ceb7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e12827ad93c47cf9013e238215ceb7a already in progress: creating tablet
I20260926 01:55:56.877767 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:55:56.896447 9194 tablet_service.cc:1511] Processing CreateTablet for tablet 215a061440854c56b00df9fda9192598 (DEFAULT_TABLE table=default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.896687 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
I20260926 01:55:56.897344 8540 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:55:56.897610 9194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215a061440854c56b00df9fda9192598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.898761 8536 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet b659eca3c97144aaacd41e747aecd96d
I20260926 01:55:56.900354 8536 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 238) took 1748 ms. Trace:
I20260926 01:55:56.902169 8538 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576] alter complete (version 1)
I20260926 01:55:56.904145 9193 tablet_service.cc:1511] Processing CreateTablet for tablet 8711788a85f94c599224123f6b8cc814 (DEFAULT_TABLE table=default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.904469 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_4e4c17faa0a4400d969f7de29bf7dcee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:56.905962 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e4c17faa0a4400d969f7de29bf7dcee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:56.900652 8536 rpcz_store.cc:276] 0926 01:55:55.152377 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.391736 (+239359us) service_pool.cc:224] Handling call
0926 01:55:55.394540 (+ 2804us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607876 00007f717c1b6b34 00007f717c1b6ac2 00007f717c1b69f1 00007f717c1b696c 00007f717c1b5c22 00007f717a45c09c 00007f717a45b4d6
0926 01:55:55.395625 (+ 1085us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:55.569789 (+174164us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:55.570009 (+ 220us) catalog_manager.cc:2863] Locking tablets
0926 01:55:55.647880 (+ 77871us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:56.435409 (+787529us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:56.435641 (+ 232us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:56.898659 (+463018us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:56.900286 (+ 1627us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:56.252027 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:56.252439 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:56.252463 (+ 24us) write_op.cc:435] Acquired schema lock
0926 01:55:56.252474 (+ 11us) tablet.cc:662] Decoding operations
0926 01:55:56.252834 (+ 360us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:56.252879 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:56.252903 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:56.253179 (+ 276us) tablet.cc:701] Row locks acquired
0926 01:55:56.253193 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:56.253317 (+ 124us) write_op.cc:270] Start()
0926 01:55:56.253413 (+ 96us) write_op.cc:276] Timestamp: P: 1790387756253247 usec, L: 0
0926 01:55:56.253428 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:56.253981 (+ 553us) log.cc:844] Serialized 768 byte log entry
0926 01:55:56.431267 (+177286us) op_driver.cc:464] REPLICATION: finished
0926 01:55:56.432605 (+ 1338us) write_op.cc:301] APPLY: starting
0926 01:55:56.432676 (+ 71us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:56.432798 (+ 122us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:56.432916 (+ 118us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:56.433209 (+ 293us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:56.433354 (+ 145us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:56.433403 (+ 49us) write_op.cc:312] APPLY: finished
0926 01:55:56.433598 (+ 195us) log.cc:844] Serialized 29 byte log entry
0926 01:55:56.433769 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.433852 (+ 83us) write_op.cc:454] Released schema lock
0926 01:55:56.433887 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:56.434145 (+ 258us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.434170 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":70,"spinlock_wait_cycles":4396416,"child_traces":[["op",{"apply.queue_time_us":1155,"apply.run_cpu_time_us":1503,"apply.run_wall_time_us":1502,"num_ops":2,"prepare.queue_time_us":601965,"prepare.run_cpu_time_us":2309,"prepare.run_wall_time_us":48414,"raft.queue_time_us":108,"raft.run_cpu_time_us":807,"raft.run_wall_time_us":1280,"replication_time_us":177729}]]}
I20260926 01:55:56.909492 9193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8711788a85f94c599224123f6b8cc814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.913192 9917 raft_consensus.cc:493] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.912891 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:56.913671 9917 raft_consensus.cc:515] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:56.933144 8538 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 41) took 2296 ms (client timeout 1000 ms). Trace:
I20260926 01:55:56.944161 9891 raft_consensus.cc:493] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:56.944672 9891 raft_consensus.cc:515] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:56.951607 9891 leader_election.cc:290] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:56.953521 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:55:56.933517 8538 rpcz_store.cc:269] 0926 01:55:54.637229 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.637386 (+ 157us) service_pool.cc:224] Handling call
0926 01:55:54.646955 (+ 9569us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:56.933038 (+2286083us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:55.114584 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:55.114948 (+ 364us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:55.114972 (+ 24us) write_op.cc:435] Acquired schema lock
0926 01:55:55.114985 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:55.115267 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:55.115306 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:55.115330 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:55.115482 (+ 152us) tablet.cc:701] Row locks acquired
0926 01:55:55.115497 (+ 15us) write_op.cc:260] PREPARE: finished
0926 01:55:55.115618 (+ 121us) write_op.cc:270] Start()
0926 01:55:55.115710 (+ 92us) write_op.cc:276] Timestamp: P: 1790387755115568 usec, L: 0
0926 01:55:55.115727 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:55:55.116250 (+ 523us) log.cc:844] Serialized 356 byte log entry
0926 01:55:55.428838 (+312588us) op_driver.cc:464] REPLICATION: finished
0926 01:55:55.433996 (+ 5158us) write_op.cc:301] APPLY: starting
0926 01:55:55.434058 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:55.434202 (+ 144us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:55.434294 (+ 92us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:55.434510 (+ 216us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:55.434647 (+ 137us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:55.434688 (+ 41us) write_op.cc:312] APPLY: finished
0926 01:55:55.434895 (+ 207us) log.cc:844] Serialized 23 byte log entry
0926 01:55:55.435063 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:55.435146 (+ 83us) write_op.cc:454] Released schema lock
0926 01:55:55.435200 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:55.435432 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:55.435455 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0926 01:55:55.905759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:55.906182 (+ 423us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:55.906205 (+ 23us) write_op.cc:435] Acquired schema lock
0926 01:55:55.906219 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:55.906525 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:55.906569 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:55.906592 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:55.906769 (+ 177us) tablet.cc:701] Row locks acquired
0926 01:55:55.906781 (+ 12us) write_op.cc:260] PREPARE: finished
0926 01:55:55.906891 (+ 110us) write_op.cc:270] Start()
0926 01:55:55.906979 (+ 88us) write_op.cc:276] Timestamp: P: 1790387755906820 usec, L: 0
0926 01:55:55.906993 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:55.907518 (+ 525us) log.cc:844] Serialized 356 byte log entry
0926 01:55:55.917197 (+ 9679us) op_driver.cc:464] REPLICATION: finished
0926 01:55:55.917569 (+ 372us) write_op.cc:301] APPLY: starting
0926 01:55:55.917649 (+ 80us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:55.917752 (+ 103us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:55.917867 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:55.918051 (+ 184us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:55.918194 (+ 143us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:55.918232 (+ 38us) write_op.cc:312] APPLY: finished
0926 01:55:55.918620 (+ 388us) log.cc:844] Serialized 23 byte log entry
0926 01:55:55.919336 (+ 716us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:55.919407 (+ 71us) write_op.cc:454] Released schema lock
0926 01:55:55.919439 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:56.006666 (+ 87227us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:56.006694 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":951,"reported_tablets":7,"spinlock_wait_cycles":14505728,"child_traces":[["op",{"apply.queue_time_us":4894,"apply.run_cpu_time_us":1408,"apply.run_wall_time_us":1408,"num_ops":1,"prepare.queue_time_us":238429,"prepare.run_cpu_time_us":1984,"prepare.run_wall_time_us":45054,"replication_time_us":312905}],["op",{"apply.queue_time_us":221,"apply.run_cpu_time_us":1406,"apply.run_wall_time_us":2224,"num_ops":1,"prepare.queue_time_us":332856,"prepare.run_cpu_time_us":2097,"prepare.run_wall_time_us":100878,"raft.queue_time_us":94,"raft.run_cpu_time_us":787,"raft.run_wall_time_us":2106,"replication_time_us":10098}]]}
I20260926 01:55:56.964407 9917 leader_election.cc:290] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:56.966068 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:56.967625 9152 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:56.975226 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet b659eca3c97144aaacd41e747aecd96d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:55 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:56.975999 9710 tablet_replica.cc:333] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:56.976752 9710 raft_consensus.cc:2243] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:56.976890 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 030c41a9718d47ddaa33382b20a60dab (DEFAULT_TABLE table=default.Table_b9e652042f524ac7973179d178f9fb52 [id=bf6a56161a714ed8abb3f65b5770813a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.977273 9710 raft_consensus.cc:2272] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:56.977977 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030c41a9718d47ddaa33382b20a60dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.978941 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:56.979212 9710 ts_tablet_manager.cc:1916] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:56.979583 9073 raft_consensus.cc:2468] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:56.981140 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:56.981681 9073 raft_consensus.cc:2468] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:56.984706 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d56f3a5dfed4ac78f1da92e22db558a with delete_type TABLET_DATA_DELETED (replaced by tablet 1c5c66728bb54ec6beb4f0a2c25d9105) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:56.986367 9050 tablet_service.cc:1511] Processing CreateTablet for tablet e140cf11b8bb49f9a94c0158a2afca4d (DEFAULT_TABLE table=default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.987434 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e140cf11b8bb49f9a94c0158a2afca4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.989547 9192 tablet_service.cc:1511] Processing CreateTablet for tablet e7512e1d08e74ef6a2c699352347eb8d (DEFAULT_TABLE table=default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.990276 9200 tablet_service.cc:1511] Processing CreateTablet for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (DEFAULT_TABLE table=default.Table_3478e66c0ff74bc4a27eb418e346e4e2 [id=05847c8c0e744754b5456a6773f5e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:56.990692 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7512e1d08e74ef6a2c699352347eb8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.991294 9200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0558d37a3b5f4bcea43a8da7b99ded09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:56.999886 9710 ts_tablet_manager.cc:1929] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:57.000293 9710 log.cc:1199] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/b659eca3c97144aaacd41e747aecd96d
I20260926 01:55:57.000360 9191 tablet_service.cc:1511] Processing CreateTablet for tablet 89fb965257de4db6b043b4a6dbe22709 (DEFAULT_TABLE table=default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.001006 9710 ts_tablet_manager.cc:1950] T b659eca3c97144aaacd41e747aecd96d P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:57.001681 9191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fb965257de4db6b043b4a6dbe22709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.008646 9190 tablet_service.cc:1511] Processing CreateTablet for tablet b612e8955d5f42dd8658320031fe36ca (DEFAULT_TABLE table=default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.009838 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b612e8955d5f42dd8658320031fe36ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.009547 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_75d76cb423f64e279ac8c55a99a7b026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:57.011245 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75d76cb423f64e279ac8c55a99a7b026 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:56.967360 8533 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 42) took 1254 ms (client timeout 1000 ms). Trace:
W20260926 01:55:57.014559 8533 rpcz_store.cc:269] 0926 01:55:55.713277 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.725331 (+ 12054us) service_pool.cc:224] Handling call
0926 01:55:56.967258 (+1241927us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7978,"reported_tablets":14}
W20260926 01:55:56.971727 8539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 32) took 1956 ms (client timeout 1000 ms). Trace:
W20260926 01:55:57.016103 8539 rpcz_store.cc:269] 0926 01:55:55.015358 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.015669 (+ 311us) service_pool.cc:224] Handling call
0926 01:55:55.026001 (+ 10332us) spinlock_profiling.cc:243] Waited 6.36 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:56.971646 (+1945645us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4527,"reported_tablets":21,"spinlock_wait_cycles":14617856}
W20260926 01:55:57.006754 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:57.033911 9045 tablet_service.cc:1511] Processing CreateTablet for tablet ec749c375d784786ba93020996708bb1 (DEFAULT_TABLE table=default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.035288 9045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec749c375d784786ba93020996708bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.054832 9829 raft_consensus.cc:493] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:57.055351 9829 raft_consensus.cc:515] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:57.055899 9189 tablet_service.cc:1511] Processing CreateTablet for tablet 8d198e51b88948548cfd95dd893a26ef (DEFAULT_TABLE table=default.Table_e3f193c692be4f919e17e315a529b202 [id=2ce052b956be496abee7b149d1e9663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.057199 9189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d198e51b88948548cfd95dd893a26ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.065117 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:57.065732 9070 raft_consensus.cc:2468] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:57.067907 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:57.068445 9070 raft_consensus.cc:2468] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:57.070621 9155 leader_election.cc:304] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:57.072571 9858 consensus_queue.cc:1048] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:57.058251 9829 leader_election.cc:290] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:57.073889 9858 raft_consensus.cc:2804] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:57.074208 9858 raft_consensus.cc:493] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:57.074599 9858 raft_consensus.cc:3060] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:57.074777 8539 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_5bb843a35ff742499a87f56bb5d77d32" } modify_external_catalogs: true
I20260926 01:55:57.075484 8539 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_5bb843a35ff742499a87f56bb5d77d32" } modify_external_catalogs: true
I20260926 01:55:57.080322 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 2c0cd3e30f084966b27921e658cacf58 (DEFAULT_TABLE table=default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.081482 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0cd3e30f084966b27921e658cacf58. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:56.887231 9339 ts_tablet_manager.cc:1469] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Trace:
0926 01:55:50.638094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:50.638240 (+ 146us) service_pool.cc:224] Handling call
0926 01:55:50.639713 (+ 1473us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:50.639797 (+ 84us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:50.677971 (+ 38174us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:50.764116 (+ 86145us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:50.764525 (+ 409us) inbound_call.cc:177] Queueing success response
0926 01:55:55.839628 (+5075103us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0926 01:55:56.366691 (+527063us) ts_tablet_manager.cc:1435] Starting tablet replica
0926 01:55:56.481727 (+115036us) tablet_replica.cc:261] Starting instrumentation
0926 01:55:56.482232 (+ 505us) tablet_replica.cc:280] Starting consensus
0926 01:55:56.720046 (+237814us) spinlock_profiling.cc:243] Waited 8.64 ms on lock 0x7b4c00002f40. stack: 00007f0e62947323 00007f0e62947220 00007f0e624eb574 0000000000516964 00000000005168f9 00007f0e67739fe0 00007f0e6774afc9 00007f0e67710234 00007f0e677384ed 00007f0e67738153 00007f0e69ad694b 00007f0e69ad7475 00007f0e69ad26f0 00007f0e69ad1400 00007f0e69f23d3e 00007f0e6ab8c3ba
Metrics: {"mutex_wait_us":78201,"raft.queue_time_us":13030,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"spinlock_wait_cycles":19863424,"tablet-open.queue_time_us":5075150}
I20260926 01:55:57.097936 9044 tablet_service.cc:1511] Processing CreateTablet for tablet 8711788a85f94c599224123f6b8cc814 (DEFAULT_TABLE table=default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.105873 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_85df772c6e434f49b9b7b7c88949d303" } new_table_name: "default.Table_7fe57989dc2540bca5e9adf318b41015" modify_external_catalogs: true
I20260926 01:55:57.107977 9044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8711788a85f94c599224123f6b8cc814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.118870 9183 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:55:57.119809 9047 consensus_meta.cc:287] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.514s user 0.006s sys 0.003s
I20260926 01:55:57.123940 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e12827ad93c47cf9013e238215ceb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:56 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:57.126906 9183 tablet_service.cc:1558] Processing DeleteTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.128321 9183 tablet_service.cc:1558] Processing DeleteTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab082379974486bbea9f87f5aa64e10) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.082065 9858 raft_consensus.cc:515] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:57.132580 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:57.134364 9152 leader_election.cc:343] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:57.142614 8883 tablet_service.cc:1511] Processing CreateTablet for tablet e140cf11b8bb49f9a94c0158a2afca4d (DEFAULT_TABLE table=default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.143906 8883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e140cf11b8bb49f9a94c0158a2afca4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.154542 9046 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51458 (request call id 191) took 1020 ms. Trace:
I20260926 01:55:57.170264 8879 tablet_service.cc:1511] Processing CreateTablet for tablet ec749c375d784786ba93020996708bb1 (DEFAULT_TABLE table=default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.171612 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec749c375d784786ba93020996708bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.182258 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 2c0cd3e30f084966b27921e658cacf58 (DEFAULT_TABLE table=default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.183485 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0cd3e30f084966b27921e658cacf58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.154935 9046 rpcz_store.cc:276] 0926 01:55:56.134574 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.134777 (+ 203us) service_pool.cc:224] Handling call
0926 01:55:56.510442 (+375665us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:56.542412 (+ 31970us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:56.659195 (+116783us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.125379 (+466184us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:57.154439 (+ 29060us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8576}
I20260926 01:55:57.211844 9047 tablet_service.cc:1511] Processing CreateTablet for tablet 215a061440854c56b00df9fda9192598 (DEFAULT_TABLE table=default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.212984 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215a061440854c56b00df9fda9192598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.214870 9339 tablet_bootstrap.cc:492] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:57.218987 9052 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51458 (request call id 193) took 1081 ms. Trace:
I20260926 01:55:57.219350 9052 rpcz_store.cc:276] 0926 01:55:56.135292 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.136879 (+ 1587us) service_pool.cc:224] Handling call
0926 01:55:56.485124 (+348245us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:56.485237 (+ 113us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:56.623925 (+138688us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.123467 (+499542us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:57.216268 (+ 92801us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:57.225514 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet b659eca3c97144aaacd41e747aecd96d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:55 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:57.227164 9341 tablet_bootstrap.cc:654] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:57.241505 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 030c41a9718d47ddaa33382b20a60dab (DEFAULT_TABLE table=default.Table_b9e652042f524ac7973179d178f9fb52 [id=bf6a56161a714ed8abb3f65b5770813a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.241746 9053 tablet_service.cc:1511] Processing CreateTablet for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (DEFAULT_TABLE table=default.Table_3478e66c0ff74bc4a27eb418e346e4e2 [id=05847c8c0e744754b5456a6773f5e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.243468 9049 tablet_service.cc:1511] Processing CreateTablet for tablet e7512e1d08e74ef6a2c699352347eb8d (DEFAULT_TABLE table=default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.245162 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 8d198e51b88948548cfd95dd893a26ef (DEFAULT_TABLE table=default.Table_e3f193c692be4f919e17e315a529b202 [id=2ce052b956be496abee7b149d1e9663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.247402 9053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0558d37a3b5f4bcea43a8da7b99ded09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.249215 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7512e1d08e74ef6a2c699352347eb8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.252164 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d198e51b88948548cfd95dd893a26ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.261809 9043 tablet_service.cc:1511] Processing CreateTablet for tablet 6752df6a58914df5b8718dd0439aa885 (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.263986 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030c41a9718d47ddaa33382b20a60dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.265302 9711 tablet_replica.cc:333] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:57.266664 9050 tablet_service.cc:1511] Processing CreateTablet for tablet b612e8955d5f42dd8658320031fe36ca (DEFAULT_TABLE table=default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.267807 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b612e8955d5f42dd8658320031fe36ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.270346 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 89fb965257de4db6b043b4a6dbe22709 (DEFAULT_TABLE table=default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.278296 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fb965257de4db6b043b4a6dbe22709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.292858 9711 raft_consensus.cc:2243] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:57.311026 9711 raft_consensus.cc:2272] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:57.311252 9339 tablet_bootstrap.cc:654] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:57.389161 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e12827ad93c47cf9013e238215ceb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:56 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:57.390403 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 8711788a85f94c599224123f6b8cc814 (DEFAULT_TABLE table=default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.390869 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 215a061440854c56b00df9fda9192598 (DEFAULT_TABLE table=default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.391199 8886 tablet_service.cc:1558] Processing DeleteTablet for tablet b659eca3c97144aaacd41e747aecd96d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:55 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:57.391547 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8711788a85f94c599224123f6b8cc814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.392118 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215a061440854c56b00df9fda9192598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.392383 8886 tablet_service.cc:1511] Processing CreateTablet for tablet 030c41a9718d47ddaa33382b20a60dab (DEFAULT_TABLE table=default.Table_b9e652042f524ac7973179d178f9fb52 [id=bf6a56161a714ed8abb3f65b5770813a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.393435 8886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030c41a9718d47ddaa33382b20a60dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.394394 8887 tablet_service.cc:1511] Processing CreateTablet for tablet 89fb965257de4db6b043b4a6dbe22709 (DEFAULT_TABLE table=default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.395123 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (DEFAULT_TABLE table=default.Table_3478e66c0ff74bc4a27eb418e346e4e2 [id=05847c8c0e744754b5456a6773f5e41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.395521 8887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fb965257de4db6b043b4a6dbe22709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.396009 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0558d37a3b5f4bcea43a8da7b99ded09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.398061 8878 tablet_service.cc:1511] Processing CreateTablet for tablet e7512e1d08e74ef6a2c699352347eb8d (DEFAULT_TABLE table=default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.398975 8878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7512e1d08e74ef6a2c699352347eb8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.415527 9858 leader_election.cc:290] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:57.421377 9340 ts_tablet_manager.cc:1434] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.653s user 0.006s sys 0.001s
W20260926 01:55:57.421875 9340 ts_tablet_manager.cc:1467] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: Tablet startup took 653ms
W20260926 01:55:57.422267 9340 ts_tablet_manager.cc:1469] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: Trace:
0926 01:55:48.941696 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:49.037924 (+ 96228us) service_pool.cc:224] Handling call
0926 01:55:49.067419 (+ 29495us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:49.067517 (+ 98us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:49.305550 (+238033us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:49.342571 (+ 37021us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:49.342977 (+ 406us) inbound_call.cc:177] Queueing success response
0926 01:55:56.398059 (+7055082us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0926 01:55:56.768025 (+369966us) ts_tablet_manager.cc:1435] Starting tablet replica
0926 01:55:56.799422 (+ 31397us) tablet_replica.cc:261] Starting instrumentation
0926 01:55:56.805621 (+ 6199us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":139,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":183,"tablet-open.queue_time_us":7055169}
I20260926 01:55:57.439620 9183 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.440995 9183 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 with delete_type TABLET_DATA_DELETED (replaced by tablet b612e8955d5f42dd8658320031fe36ca) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.444792 9805 raft_consensus.cc:493] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:57.445287 9805 raft_consensus.cc:515] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:57.446341 9200 tablet_service.cc:1558] Processing DeleteTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.447319 9805 leader_election.cc:290] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:57.463389 9981 tablet.cc:1751] T e53c9eb8a6d648b2a9359f02b7f8764b P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260926 01:55:57.474979 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 404423us
I20260926 01:55:57.476030 9155 leader_election.cc:304] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
W20260926 01:55:57.476848 9155 raft_consensus.cc:2686] T 9bac55da5b164c0f981914d39c87c33e P bc00feec3f564608851f0112e97ab532: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 01:55:57.479135 9155 leader_election.cc:304] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
W20260926 01:55:57.498799 8624 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.007s sys 0.003s
W20260926 01:55:57.499243 8624 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.645s user 0.007s sys 0.003s
I20260926 01:55:57.505916 10016 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:55:57.418115 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 03086ec5451743fdbf2d5726d43f31b1 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]) successfully deleted
W20260926 01:55:57.507076 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 88977.7us
I20260926 01:55:57.507401 10016 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:57.507781 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet b659eca3c97144aaacd41e747aecd96d (table default.Table_ee6cc37e9698474a8a3194f060b8c028 [id=0dae284524224264b7ccbd59cd975030]) successfully deleted
W20260926 01:55:57.508461 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 75d4598596ff49378fc3536fff0ac1b1 because the tablet was not found. No further retry: Not found: Tablet not found: 75d4598596ff49378fc3536fff0ac1b1
W20260926 01:55:57.509049 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 55452e70a7fe48efbafdc7cbb29c6272 because the tablet was not found. No further retry: Not found: Tablet not found: 55452e70a7fe48efbafdc7cbb29c6272
W20260926 01:55:57.509523 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 03086ec5451743fdbf2d5726d43f31b1 because the tablet was not found. No further retry: Not found: Tablet not found: 03086ec5451743fdbf2d5726d43f31b1
I20260926 01:55:57.532292 10016 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
W20260926 01:55:57.560272 8599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
I20260926 01:55:56.828511 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:57.573024 8843 consensus_peers.cc:597] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b659eca3c97144aaacd41e747aecd96d. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:56.853737 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 188) took 2411 ms. Trace:
I20260926 01:55:57.536276 8880 tablet_service.cc:1511] Processing CreateTablet for tablet b612e8955d5f42dd8658320031fe36ca (DEFAULT_TABLE table=default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.537175 8885 tablet_service.cc:1558] Processing DeleteTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:57.575147 9812 rpcz_store.cc:276] 0926 01:55:54.442187 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.445536 (+ 3349us) service_pool.cc:224] Handling call
0926 01:55:56.553744 (+2108208us) ts_tablet_manager.cc:1151] Deleting tablet 1df1d4daaddc4ac4aa02aa7763ea3184
0926 01:55:56.553799 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:56.629651 (+ 75852us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.740146 (+110495us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:56.853661 (+113515us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2106947}
I20260926 01:55:57.537988 8877 tablet_service.cc:1511] Processing CreateTablet for tablet 8d198e51b88948548cfd95dd893a26ef (DEFAULT_TABLE table=default.Table_e3f193c692be4f919e17e315a529b202 [id=2ce052b956be496abee7b149d1e9663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.539497 8876 tablet_service.cc:1558] Processing DeleteTablet for tablet 03086ec5451743fdbf2d5726d43f31b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab082379974486bbea9f87f5aa64e10) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:57.542236 8875 tablet_service.cc:1511] Processing CreateTablet for tablet 6752df6a58914df5b8718dd0439aa885 (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.544469 8874 tablet_service.cc:1511] Processing CreateTablet for tablet a6e8059b94564c42894c6c8a3d695950 (DEFAULT_TABLE table=default.Table_42a00bb5a7f0447e9f894eb5d3019426 [id=d680dfae1d21415382be575ea6696a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.545252 8873 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:57.546268 8872 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:57.547421 8871 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:58226
W20260926 01:55:57.629160 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143): Network error: Client connection negotiation failed: client connection to 127.8.76.188:38143: connect: Connection refused (error 111)
W20260926 01:55:57.485373 9216 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.024s sys 0.049s
W20260926 01:55:57.652442 9216 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.839s user 0.024s sys 0.049s
I20260926 01:55:57.495910 9188 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.516793 9184 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.517746 9185 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:57.654134 9710 tablet_replica.cc:333] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:55:57.522491 9186 tablet_service.cc:1511] Processing CreateTablet for tablet a6e8059b94564c42894c6c8a3d695950 (DEFAULT_TABLE table=default.Table_42a00bb5a7f0447e9f894eb5d3019426 [id=d680dfae1d21415382be575ea6696a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.654984 9710 raft_consensus.cc:2243] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:57.655242 9186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e8059b94564c42894c6c8a3d695950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.655980 9710 raft_consensus.cc:2272] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:57.658207 9710 ts_tablet_manager.cc:1916] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:57.544909 10024 tablet.cc:1751] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260926 01:55:57.658871 9989 raft_consensus.cc:2804] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260926 01:55:57.545387 9197 consensus_meta.cc:287] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.585s user 0.007s sys 0.000s
I20260926 01:55:57.659202 9989 raft_consensus.cc:493] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:57.659492 9989 raft_consensus.cc:3060] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:57.598436 9198 consensus_meta.cc:287] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.556s user 0.007s sys 0.000s
I20260926 01:55:57.602855 9341 tablet_bootstrap.cc:492] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
W20260926 01:55:57.592954 9191 consensus_meta.cc:287] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.000s
I20260926 01:55:57.678778 9341 ts_tablet_manager.cc:1403] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.918s user 0.011s sys 0.000s
I20260926 01:55:57.680029 9989 raft_consensus.cc:515] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:57.670243 9048 consensus_meta.cc:287] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.641s user 0.007s sys 0.001s
I20260926 01:55:57.671114 9043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6752df6a58914df5b8718dd0439aa885. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:57.686447 9003 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.005s sys 0.013s
W20260926 01:55:57.673350 9045 consensus_meta.cc:287] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.569s user 0.006s sys 0.002s
W20260926 01:55:57.689795 9003 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.859s user 0.006s sys 0.013s
I20260926 01:55:57.723877 9048 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51458 (request call id 194) took 1585 ms. Trace:
I20260926 01:55:57.724327 9048 rpcz_store.cc:276] 0926 01:55:56.138267 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.138484 (+ 217us) service_pool.cc:224] Handling call
0926 01:55:56.476845 (+338361us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b5800000f98. stack: 00007f0e62947323 00007f0e62947220 00007f0e624eb574 0000000000516964 00000000005168f9 00007f0e627fd8a1 00007f0e62941e7c 00007f0e62941a3b 00007f0e6b88c119 00007f0e6aac6a70 00007f0e6ababe61 00007f0e6ab890f3 00007f0e6ab37640 00007f0e690c1bf4 00007f0e690c1b82 00007f0e690c1ab1
0926 01:55:56.555238 (+ 78393us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:56.561006 (+ 5768us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:56.918361 (+357355us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.715349 (+796988us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:57.723798 (+ 8449us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35761408}
I20260926 01:55:57.726960 9711 ts_tablet_manager.cc:1916] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:57.736505 8882 consensus_meta.cc:287] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510: Time spent flushing consensus metadata: real 0.543s user 0.005s sys 0.000s
W20260926 01:55:57.510274 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 39cd9130640d4b14beef42e9f1b36b99 because the tablet was not found. No further retry: Not found: Tablet not found: 39cd9130640d4b14beef42e9f1b36b99
W20260926 01:55:57.737886 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 227612us
W20260926 01:55:57.761960 8883 consensus_meta.cc:287] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510: Time spent flushing consensus metadata: real 0.603s user 0.006s sys 0.001s
I20260926 01:55:57.767133 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 1df1d4daaddc4ac4aa02aa7763ea3184 (table default.Table_d45fa6bcc6b14d1984b1e756819371a4 [id=dfd1961813da435d8cf918532748462b]) successfully deleted
I20260926 01:55:57.767835 8877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d198e51b88948548cfd95dd893a26ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.684214 9341 raft_consensus.cc:359] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:57.769886 9341 raft_consensus.cc:385] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:57.770233 9341 raft_consensus.cc:740] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:57.771221 9341 consensus_queue.cc:260] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:57.772043 8875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6752df6a58914df5b8718dd0439aa885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.773473 9341 ts_tablet_manager.cc:1434] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.093s user 0.005s sys 0.000s
I20260926 01:55:57.775134 9341 tablet_bootstrap.cc:492] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:57.779698 9812 tablet_replica.cc:333] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:57.783370 9341 tablet_bootstrap.cc:654] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:57.790884 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b612e8955d5f42dd8658320031fe36ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.804538 9812 raft_consensus.cc:2243] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:57.804688 8531 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 250) took 1448 ms. Trace:
I20260926 01:55:57.805225 9812 raft_consensus.cc:2272] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:57.805027 8531 rpcz_store.cc:276] 0926 01:55:56.356017 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.879058 (+523041us) service_pool.cc:224] Handling call
0926 01:55:57.804606 (+925548us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:57.806100 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:57.807232 9812 ts_tablet_manager.cc:1916] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:57.807821 9989 leader_election.cc:290] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:57.812252 8537 catalog_manager.cc:5838] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260926 01:55:57.813939 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:55:57.814857 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:55:57.815659 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 4 got 3
I20260926 01:55:57.816450 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:57.819995 9340 tablet_bootstrap.cc:492] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:57.820467 9041 tablet_service.cc:1511] Processing CreateTablet for tablet a6e8059b94564c42894c6c8a3d695950 (DEFAULT_TABLE table=default.Table_42a00bb5a7f0447e9f894eb5d3019426 [id=d680dfae1d21415382be575ea6696a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:57.821671 9041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e8059b94564c42894c6c8a3d695950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:57.833801 8874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6e8059b94564c42894c6c8a3d695950. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:55:57.846372 9049 consensus_meta.cc:287] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.523s user 0.004s sys 0.004s
W20260926 01:55:57.852180 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 681900e0fac74cdabaea1d8a7fb4a189 consensus state has no leader
I20260926 01:55:57.851840 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 281 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:55:57.854853 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 70ef5265eb6244aa96c8195aa30cdfb0 because the tablet was not found. No further retry: Not found: Tablet not found: 70ef5265eb6244aa96c8195aa30cdfb0
W20260926 01:55:57.855710 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet c27f1dd7bb0449a6a3fa371a8636e09b because the tablet was not found. No further retry: Not found: Tablet not found: c27f1dd7bb0449a6a3fa371a8636e09b
I20260926 01:55:57.857896 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:55:57.858857 10016 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260926 01:55:57.865437 9339 tablet_bootstrap.cc:492] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:57.866007 9339 ts_tablet_manager.cc:1403] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.651s user 0.012s sys 0.000s
I20260926 01:55:57.866044 9940 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51458 (request call id 196) took 1726 ms. Trace:
I20260926 01:55:57.869102 9339 raft_consensus.cc:359] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:57.869781 9339 raft_consensus.cc:385] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:57.870044 9339 raft_consensus.cc:740] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:57.875290 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3571967749844fd7859cf179a93cf403" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:57.876041 9073 raft_consensus.cc:2393] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:57.875744 9339 consensus_queue.cc:260] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:57.877460 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:57.877826 9339 ts_tablet_manager.cc:1434] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260926 01:55:57.878794 9339 tablet_bootstrap.cc:492] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:57.879971 9340 tablet_bootstrap.cc:654] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:57.885037 9006 leader_election.cc:304] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:57.885883 10032 raft_consensus.cc:2804] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:57.886286 10032 raft_consensus.cc:493] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:57.886718 10032 raft_consensus.cc:3060] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:57.893589 9339 tablet_bootstrap.cc:654] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:57.895530 9349 thread.cc:641] excess-log-deleter (server) Time spent creating pthread: real 0.592s user 0.166s sys 0.240s
W20260926 01:55:57.895865 9349 thread.cc:608] excess-log-deleter (server) Time spent starting thread: real 0.593s user 0.166s sys 0.240s
W20260926 01:55:57.897909 8540 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 33) took 1839 ms (client timeout 1000 ms). Trace:
I20260926 01:55:57.913170 9812 ts_tablet_manager.cc:1929] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:57.913650 9812 log.cc:1199] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/c27f1dd7bb0449a6a3fa371a8636e09b
I20260926 01:55:57.914436 9812 ts_tablet_manager.cc:1950] T c27f1dd7bb0449a6a3fa371a8636e09b P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:57.920666 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 190) took 3471 ms. Trace:
I20260926 01:55:57.923295 9812 rpcz_store.cc:276] 0926 01:55:54.449741 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.449958 (+ 217us) service_pool.cc:224] Handling call
0926 01:55:57.576881 (+3126923us) ts_tablet_manager.cc:1151] Deleting tablet c27f1dd7bb0449a6a3fa371a8636e09b
0926 01:55:57.779471 (+202590us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b5800000f98. stack: 00007f94aec19323 00007f94aec19220 00007f94ae7bd574 0000000000516964 00000000005168f9 00007f94aeacf8a1 00007f94aec13b51 00007f94aec14182 00007f94b7b872c9 00007f94b6d98918 00007f94b6e5f2c9 00007f94b6e5f8c3 00007f94b6e90960 00007f94b6e76e6d 00007f94b6e76e19 00007f94b6e76da9
0926 01:55:57.779538 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:57.871582 (+ 92044us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.912992 (+ 41410us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.920573 (+ 7581us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":422898432,"tablet-delete.queue_time_us":3094157}
I20260926 01:55:57.926321 9812 tablet_replica.cc:333] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:57.927294 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2ee16b0c3e84936abba8248f194786a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:57.928679 9812 raft_consensus.cc:2243] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:57.929539 9812 raft_consensus.cc:2272] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:57.929488 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:57.881536 9940 rpcz_store.cc:276] 0926 01:55:56.139773 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.140186 (+ 413us) service_pool.cc:224] Handling call
0926 01:55:57.865973 (+1725787us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:56.141764 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:55:56.142176 (+ 412us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:55:56.142230 (+ 54us) alter_schema_op.cc:66] Acquired schema lock
0926 01:55:56.142317 (+ 87us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:55:56.142472 (+ 155us) alter_schema_op.cc:128] START. Timestamp: 1790387756142330
0926 01:55:56.142499 (+ 27us) op_driver.cc:348] REPLICATION: starting
0926 01:55:56.143131 (+ 632us) log.cc:844] Serialized 173 byte log entry
0926 01:55:56.186142 (+ 43011us) op_driver.cc:464] REPLICATION: finished
0926 01:55:56.454199 (+268057us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:55:56.604462 (+150263us) log.cc:844] Serialized 5 byte log entry
0926 01:55:57.214067 (+609605us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.659754 (+445687us) log.cc:844] Serialized 14 byte log entry
0926 01:55:57.865800 (+206046us) alter_schema_op.cc:73] Released schema lock
0926 01:55:57.865832 (+ 32us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":257867,"prepare.queue_time_us":164,"prepare.run_cpu_time_us":7199,"prepare.run_wall_time_us":22300,"replication_time_us":43498,"spinlock_wait_cycles":1729920,"thread_start_us":408269,"threads_started":3,"wal-append.queue_time_us":220259,"wal-append.run_cpu_time_us":1842,"wal-append.run_wall_time_us":1440164}]]}
I20260926 01:55:57.933522 9812 ts_tablet_manager.cc:1916] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:57.934581 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 49616.6us
I20260926 01:55:57.937517 9340 tablet_bootstrap.cc:492] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:57.938022 9340 ts_tablet_manager.cc:1403] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.118s user 0.019s sys 0.003s
I20260926 01:55:57.940646 9340 raft_consensus.cc:359] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:57.942756 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet c27f1dd7bb0449a6a3fa371a8636e09b (table default.Table_29ce250e9d9a4162966731d01a8e3b23 [id=e664d2aa0cf74dfbad934ad658e8ec0e]) successfully deleted
I20260926 01:55:57.947288 8532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 249) took 1665 ms. Trace:
W20260926 01:55:57.601002 9152 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260926 01:55:57.947623 8532 rpcz_store.cc:276] 0926 01:55:56.282079 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.771587 (+489508us) service_pool.cc:224] Handling call
0926 01:55:56.872298 (+100711us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:55:56.930042 (+ 57744us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:55:57.946634 (+1016592us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:55:57.946928 (+ 294us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:55:57.947217 (+ 289us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:56.932624 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:56.932974 (+ 350us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:56.933000 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:55:56.933013 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:56.933311 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:56.933349 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:56.933372 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:56.933580 (+ 208us) tablet.cc:701] Row locks acquired
0926 01:55:56.933595 (+ 15us) write_op.cc:260] PREPARE: finished
0926 01:55:56.933712 (+ 117us) write_op.cc:270] Start()
0926 01:55:56.933799 (+ 87us) write_op.cc:276] Timestamp: P: 1790387756933685 usec, L: 0
0926 01:55:56.933814 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:56.934370 (+ 556us) log.cc:844] Serialized 433 byte log entry
0926 01:55:57.788216 (+853846us) op_driver.cc:464] REPLICATION: finished
0926 01:55:57.793995 (+ 5779us) write_op.cc:301] APPLY: starting
0926 01:55:57.794051 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:57.794161 (+ 110us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:57.794267 (+ 106us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:57.794752 (+ 485us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:57.913003 (+118251us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:55:57.913051 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:55:57.913264 (+ 213us) log.cc:844] Serialized 29 byte log entry
0926 01:55:57.914200 (+ 936us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:57.914275 (+ 75us) write_op.cc:454] Released schema lock
0926 01:55:57.914314 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5616,"num_ops":2,"prepare.queue_time_us":671,"prepare.run_cpu_time_us":2068,"prepare.run_wall_time_us":2068,"raft.queue_time_us":180,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":854290,"thread_start_us":78600,"threads_started":1}]]}
I20260926 01:55:57.945922 8903 raft_consensus.cc:3060] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:57.956552 9152 leader_election.cc:343] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:57.957523 8535 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 251) took 1054 ms. Trace:
I20260926 01:55:57.957844 8535 rpcz_store.cc:276] 0926 01:55:56.903332 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.904076 (+ 744us) service_pool.cc:224] Handling call
0926 01:55:56.906344 (+ 2268us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:55:56.969779 (+ 63435us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:55:57.956821 (+987042us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:55:57.957139 (+ 318us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:55:57.957451 (+ 312us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:57.163127 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:57.163571 (+ 444us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:57.163595 (+ 24us) write_op.cc:435] Acquired schema lock
0926 01:55:57.163607 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:57.163923 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:57.163969 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:57.163992 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:57.164221 (+ 229us) tablet.cc:701] Row locks acquired
0926 01:55:57.164235 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:55:57.164360 (+ 125us) write_op.cc:270] Start()
0926 01:55:57.164451 (+ 91us) write_op.cc:276] Timestamp: P: 1790387757164306 usec, L: 0
0926 01:55:57.164466 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:57.165068 (+ 602us) log.cc:844] Serialized 433 byte log entry
0926 01:55:57.867632 (+702564us) op_driver.cc:464] REPLICATION: finished
0926 01:55:57.955020 (+ 87388us) write_op.cc:301] APPLY: starting
0926 01:55:57.955101 (+ 81us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:57.955244 (+ 143us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:57.955365 (+ 121us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:57.955902 (+ 537us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:57.956064 (+ 162us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:55:57.956109 (+ 45us) write_op.cc:312] APPLY: finished
0926 01:55:57.956344 (+ 235us) log.cc:844] Serialized 29 byte log entry
0926 01:55:57.956515 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:57.956591 (+ 76us) write_op.cc:454] Released schema lock
0926 01:55:57.956625 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":87205,"mutex_wait_us":123816,"num_ops":2,"prepare.queue_time_us":144740,"prepare.run_cpu_time_us":2204,"prepare.run_wall_time_us":2206,"replication_time_us":703088,"thread_start_us":68483,"threads_started":1}]]}
I20260926 01:55:57.950654 8843 leader_election.cc:304] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:57.965540 8843 leader_election.cc:304] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:57.951511 8901 raft_consensus.cc:3060] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:57.898351 8540 rpcz_store.cc:269] 0926 01:55:56.031258 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.075990 (+ 44732us) service_pool.cc:224] Handling call
0926 01:55:56.244929 (+168939us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:57.870373 (+1625444us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:56.904517 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:56.904900 (+ 383us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:56.904925 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:55:56.904939 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:56.905208 (+ 269us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:56.905253 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:56.905276 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:56.905425 (+ 149us) tablet.cc:701] Row locks acquired
0926 01:55:56.905438 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:56.905560 (+ 122us) write_op.cc:270] Start()
0926 01:55:56.905657 (+ 97us) write_op.cc:276] Timestamp: P: 1790387756905483 usec, L: 0
0926 01:55:56.905673 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:55:56.906186 (+ 513us) log.cc:844] Serialized 530 byte log entry
0926 01:55:57.756518 (+850332us) op_driver.cc:464] REPLICATION: finished
0926 01:55:57.786339 (+ 29821us) write_op.cc:301] APPLY: starting
0926 01:55:57.786403 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:57.786518 (+ 115us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:57.786661 (+ 143us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:57.786872 (+ 211us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:57.787027 (+ 155us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:57.787070 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:55:57.787285 (+ 215us) log.cc:844] Serialized 23 byte log entry
0926 01:55:57.793304 (+ 6019us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:57.793378 (+ 74us) write_op.cc:454] Released schema lock
0926 01:55:57.793416 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:57.793713 (+ 297us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:57.793739 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":22,"spinlock_wait_cycles":376169344,"child_traces":[["op",{"apply.queue_time_us":29602,"apply.run_cpu_time_us":1568,"apply.run_wall_time_us":7350,"num_ops":1,"prepare.queue_time_us":5274,"prepare.run_cpu_time_us":1968,"prepare.run_wall_time_us":1969,"raft.queue_time_us":228,"raft.run_cpu_time_us":22,"raft.run_wall_time_us":23,"replication_time_us":850709,"thread_start_us":31253,"threads_started":1}]]}
I20260926 01:55:57.969067 8540 master_service.cc:960] ReplaceTablet: received request to replace tablet 70ef5265eb6244aa96c8195aa30cdfb0 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:57.971031 9891 raft_consensus.cc:3060] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:57.972311 9339 tablet_bootstrap.cc:492] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:57.972364 8536 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 252) took 1068 ms. Trace:
I20260926 01:55:57.972805 9339 ts_tablet_manager.cc:1403] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.094s user 0.006s sys 0.006s
I20260926 01:55:57.955665 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:57.972651 8536 rpcz_store.cc:276] 0926 01:55:56.903642 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.912478 (+ 8836us) service_pool.cc:224] Handling call
0926 01:55:57.017104 (+104626us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:55:57.018596 (+ 1492us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:55:57.971687 (+953091us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:55:57.972011 (+ 324us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:55:57.972296 (+ 285us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:57.165428 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:57.165804 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:57.165827 (+ 23us) write_op.cc:435] Acquired schema lock
0926 01:55:57.165839 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:57.166103 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:57.166138 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:57.166161 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:57.166364 (+ 203us) tablet.cc:701] Row locks acquired
0926 01:55:57.167167 (+ 803us) write_op.cc:260] PREPARE: finished
0926 01:55:57.167272 (+ 105us) write_op.cc:270] Start()
0926 01:55:57.167351 (+ 79us) write_op.cc:276] Timestamp: P: 1790387757167231 usec, L: 0
0926 01:55:57.167366 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:57.167816 (+ 450us) log.cc:844] Serialized 433 byte log entry
0926 01:55:57.953318 (+785502us) op_driver.cc:464] REPLICATION: finished
0926 01:55:57.969889 (+ 16571us) write_op.cc:301] APPLY: starting
0926 01:55:57.969951 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:57.970091 (+ 140us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:57.970226 (+ 135us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:57.970659 (+ 433us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:57.970910 (+ 251us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:55:57.970969 (+ 59us) write_op.cc:312] APPLY: finished
0926 01:55:57.971219 (+ 250us) log.cc:844] Serialized 29 byte log entry
0926 01:55:57.971387 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:57.971457 (+ 70us) write_op.cc:454] Released schema lock
0926 01:55:57.971491 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":16390,"mutex_wait_us":45064,"num_ops":2,"prepare.queue_time_us":68217,"prepare.run_cpu_time_us":1807,"prepare.run_wall_time_us":2580,"replication_time_us":785843,"threads_started":1}]]}
I20260926 01:55:57.982955 10032 raft_consensus.cc:515] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:57.984340 9340 raft_consensus.cc:385] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:57.984663 9340 raft_consensus.cc:740] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:57.985386 9340 consensus_queue.cc:260] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:57.987875 9152 leader_election.cc:343] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:57.983533 9891 raft_consensus.cc:2749] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:57.989382 9891 raft_consensus.cc:2749] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:57.999193 8532 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_b9e652042f524ac7973179d178f9fb52" } modify_external_catalogs: true
I20260926 01:55:57.999222 8843 leader_election.cc:304] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:58.000002 8532 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_b9e652042f524ac7973179d178f9fb52" } modify_external_catalogs: true
W20260926 01:55:57.743268 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 264179us
I20260926 01:55:57.689224 9710 ts_tablet_manager.cc:1929] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:58.002648 9186 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bac55da5b164c0f981914d39c87c33e with delete_type TABLET_DATA_DELETED (replaced by tablet 0558d37a3b5f4bcea43a8da7b99ded09) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.002844 9155 leader_election.cc:304] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:57.519990 9187 tablet_service.cc:1511] Processing CreateTablet for tablet 6752df6a58914df5b8718dd0439aa885 (DEFAULT_TABLE table=default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:57.608165 9195 consensus_meta.cc:287] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.659s user 0.009s sys 0.000s
W20260926 01:55:57.610085 9192 consensus_meta.cc:287] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.562s user 0.007s sys 0.001s
W20260926 01:55:57.752918 9981 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.210s user 0.000s sys 0.001s
I20260926 01:55:58.003247 9710 log.cc:1199] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:55:58.018167 10050 raft_consensus.cc:2749] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:55:58.019981 9003 leader_election.cc:343] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:58.020104 8531 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_42a00bb5a7f0447e9f894eb5d3019426" } new_table_name: "default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c" modify_external_catalogs: true
I20260926 01:55:58.020376 10028 raft_consensus.cc:493] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:58.020737 9340 ts_tablet_manager.cc:1434] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.082s user 0.003s sys 0.004s
I20260926 01:55:58.020787 10028 raft_consensus.cc:515] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.021874 9340 tablet_bootstrap.cc:492] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:55:58.021853 9003 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:58.022591 9003 consensus_peers.cc:597] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c27f1dd7bb0449a6a3fa371a8636e09b. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:58.024446 10028 leader_election.cc:290] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:58.027680 9340 tablet_bootstrap.cc:654] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:58.038702 10032 leader_election.cc:290] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:58.006343 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6752df6a58914df5b8718dd0439aa885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:58.011046 9185 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 with delete_type TABLET_DATA_DELETED (replaced by tablet b612e8955d5f42dd8658320031fe36ca) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.054217 9349 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.188:38143
I20260926 01:55:58.011512 9191 tablet_service.cc:1558] Processing DeleteTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.056928 9829 raft_consensus.cc:493] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:55:58.057468 9829 raft_consensus.cc:515] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:58.058787 10042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.188:38143 every 8 connection(s)
I20260926 01:55:58.059481 9829 leader_election.cc:290] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:58.071676 9812 ts_tablet_manager.cc:1929] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:58.072120 9812 log.cc:1199] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/70ef5265eb6244aa96c8195aa30cdfb0
I20260926 01:55:58.072856 9812 ts_tablet_manager.cc:1950] T 70ef5265eb6244aa96c8195aa30cdfb0 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:58.075361 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 191) took 3618 ms. Trace:
I20260926 01:55:58.075680 9812 rpcz_store.cc:276] 0926 01:55:54.457234 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.457370 (+ 136us) service_pool.cc:224] Handling call
0926 01:55:57.926154 (+3468784us) ts_tablet_manager.cc:1151] Deleting tablet 70ef5265eb6244aa96c8195aa30cdfb0
0926 01:55:57.926199 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.045865 (+119666us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.070535 (+ 24670us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.075297 (+ 4762us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3443368}
I20260926 01:55:58.078912 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 186) took 3650 ms. Trace:
I20260926 01:55:58.079216 9812 rpcz_store.cc:276] 0926 01:55:54.428418 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.445253 (+ 16835us) service_pool.cc:224] Handling call
0926 01:55:58.078548 (+3633295us) ts_tablet_manager.cc:1151] Deleting tablet 0c6cccff3de64f3384ce58bb84c43487
0926 01:55:58.078591 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.078851 (+ 260us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3595041}
I20260926 01:55:58.082175 9812 tablet_replica.cc:333] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
W20260926 01:55:58.082991 9006 consensus_peers.cc:597] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39cd9130640d4b14beef42e9f1b36b99. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:58.084635 9812 raft_consensus.cc:2243] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:58.085151 9812 raft_consensus.cc:2272] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:58.087289 8539 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet b4c64a7619c4452f94b8d8b4f8bbbcf8
I20260926 01:55:58.088982 8539 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 254) took 1024 ms. Trace:
I20260926 01:55:58.089308 8539 rpcz_store.cc:276] 0926 01:55:57.064765 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.074252 (+ 9487us) service_pool.cc:224] Handling call
0926 01:55:57.075991 (+ 1739us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:57.077464 (+ 1473us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:57.077612 (+ 148us) catalog_manager.cc:2863] Locking tablets
0926 01:55:57.136130 (+ 58518us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:58.086862 (+950732us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:58.087094 (+ 232us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:58.087191 (+ 97us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:58.088905 (+ 1714us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:57.169864 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:57.170209 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:57.170231 (+ 22us) write_op.cc:435] Acquired schema lock
0926 01:55:57.170244 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:57.170629 (+ 385us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:57.170665 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:57.170686 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:57.171085 (+ 399us) tablet.cc:701] Row locks acquired
0926 01:55:57.171100 (+ 15us) write_op.cc:260] PREPARE: finished
0926 01:55:57.171194 (+ 94us) write_op.cc:270] Start()
0926 01:55:57.171277 (+ 83us) write_op.cc:276] Timestamp: P: 1790387757171150 usec, L: 0
0926 01:55:57.171292 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:55:57.171711 (+ 419us) log.cc:844] Serialized 728 byte log entry
0926 01:55:57.996176 (+824465us) op_driver.cc:464] REPLICATION: finished
0926 01:55:58.076070 (+ 79894us) write_op.cc:301] APPLY: starting
0926 01:55:58.076129 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:58.076230 (+ 101us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:58.076317 (+ 87us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:58.076568 (+ 251us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:58.076696 (+ 128us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:58.076737 (+ 41us) write_op.cc:312] APPLY: finished
0926 01:55:58.076907 (+ 170us) log.cc:844] Serialized 29 byte log entry
0926 01:55:58.077525 (+ 618us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.077592 (+ 67us) write_op.cc:454] Released schema lock
0926 01:55:58.077623 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":79743,"mutex_wait_us":24814,"num_ops":2,"prepare.queue_time_us":31706,"prepare.run_cpu_time_us":1833,"prepare.run_wall_time_us":2002,"replication_time_us":824804}]]}
W20260926 01:55:58.094996 8531 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6e8059b94564c42894c6c8a3d695950 consensus state has no leader
I20260926 01:55:57.994194 9711 ts_tablet_manager.cc:1929] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:58.095700 9711 log.cc:1199] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/1cba678b66744dcfb4d13ae07ebec33f
I20260926 01:55:58.097620 9711 ts_tablet_manager.cc:1950] T 1cba678b66744dcfb4d13ae07ebec33f P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:58.100291 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 159) took 4511 ms. Trace:
I20260926 01:55:58.105358 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 70ef5265eb6244aa96c8195aa30cdfb0 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:55:58.105746 8534 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
W20260926 01:55:58.106166 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 0c6cccff3de64f3384ce58bb84c43487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c6cccff3de64f3384ce58bb84c43487 already in progress: creating tablet
I20260926 01:55:58.109989 9349 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
I20260926 01:55:58.117302 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.117996 9188 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.117970 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_a97d7498cf3d4cf18177932f26b3a1db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:58.118840 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 9349
I20260926 01:55:58.119341 9710 ts_tablet_manager.cc:1950] T 3bf2fa5fbe4146b5adb27ff09de878d3 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:55:58.119874 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a97d7498cf3d4cf18177932f26b3a1db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:58.120110 9184 tablet_service.cc:1558] Processing DeleteTablet for tablet 55452e70a7fe48efbafdc7cbb29c6272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:53 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.126436 9196 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42408 (request call id 219) took 1097 ms. Trace:
I20260926 01:55:58.127019 9196 rpcz_store.cc:276] 0926 01:55:56.976053 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.976194 (+ 141us) service_pool.cc:224] Handling call
0926 01:55:56.977518 (+ 1324us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:56.977606 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:57.156336 (+178730us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.562095 (+405759us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:58.055523 (+493428us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:58.138270 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet b659eca3c97144aaacd41e747aecd96d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:55 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.156332 10053 raft_consensus.cc:2804] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:58.157011 10053 raft_consensus.cc:493] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:58.157536 10053 raft_consensus.cc:3060] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:58.163170 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 160816us
I20260926 01:55:58.162963 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_275ba09a70cd4eaca781dec90dc353b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:58.165380 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_275ba09a70cd4eaca781dec90dc353b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:58.166075 8534 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 4 got 3
I20260926 01:55:58.167968 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_4de0847b69e04e49b345919625be76a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:58.168417 8534 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
W20260926 01:55:58.169596 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4de0847b69e04e49b345919625be76a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:58.176364 8537 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 43) took 1356 ms (client timeout 1000 ms). Trace:
W20260926 01:55:58.176828 8537 rpcz_store.cc:269] 0926 01:55:56.820412 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.879044 (+ 58632us) service_pool.cc:224] Handling call
0926 01:55:56.922406 (+ 43362us) spinlock_profiling.cc:243] Waited 6.54 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:55:58.176280 (+1253874us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:57.883433 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:57.883983 (+ 550us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:57.884012 (+ 29us) write_op.cc:435] Acquired schema lock
0926 01:55:57.884029 (+ 17us) tablet.cc:662] Decoding operations
0926 01:55:57.884334 (+ 305us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:57.884377 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:57.884403 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:57.884631 (+ 228us) tablet.cc:701] Row locks acquired
0926 01:55:57.884648 (+ 17us) write_op.cc:260] PREPARE: finished
0926 01:55:57.884791 (+ 143us) write_op.cc:270] Start()
0926 01:55:57.884893 (+ 102us) write_op.cc:276] Timestamp: P: 1790387757884712 usec, L: 0
0926 01:55:57.884913 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:55:57.997146 (+112233us) log.cc:844] Serialized 530 byte log entry
0926 01:55:57.997916 (+ 770us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c7895e8 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11 00007f717cc1cd3d 00000000005bd4e4 00000000005bd319
0926 01:55:58.072529 (+ 74613us) op_driver.cc:464] REPLICATION: finished
0926 01:55:58.090784 (+ 18255us) write_op.cc:301] APPLY: starting
0926 01:55:58.090850 (+ 66us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:58.090960 (+ 110us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:58.090984 (+ 24us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:58.091171 (+ 187us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:58.091299 (+ 128us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:58.091344 (+ 45us) write_op.cc:312] APPLY: finished
0926 01:55:58.091521 (+ 177us) log.cc:844] Serialized 23 byte log entry
0926 01:55:58.092170 (+ 649us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.092240 (+ 70us) write_op.cc:454] Released schema lock
0926 01:55:58.092278 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:58.092531 (+ 253us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.092568 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28133,"reported_tablets":10,"spinlock_wait_cycles":15043840,"child_traces":[["op",{"apply.queue_time_us":14170,"apply.run_cpu_time_us":1275,"apply.run_wall_time_us":5616,"num_ops":1,"prepare.queue_time_us":61202,"prepare.run_cpu_time_us":3951,"prepare.run_wall_time_us":114704,"raft.queue_time_us":304,"raft.run_cpu_time_us":793,"raft.run_wall_time_us":793,"replication_time_us":187516,"spinlock_wait_cycles":256323072,"thread_start_us":68548,"threads_started":1}]]}
I20260926 01:55:58.188393 8874 tablet_service.cc:1558] Processing DeleteTablet for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:57 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:58.012898 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:55:58.171030 9190 consensus_meta.cc:287] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 1.016s user 0.001s sys 0.008s
I20260926 01:55:58.198016 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:58.227186 9216 raft_consensus.cc:2468] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:58.229146 9192 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42408 (request call id 222) took 1241 ms. Trace:
I20260926 01:55:58.229557 9192 rpcz_store.cc:276] 0926 01:55:56.987979 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.988795 (+ 816us) service_pool.cc:224] Handling call
0926 01:55:56.990168 (+ 1373us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:56.990246 (+ 78us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:57.048262 (+ 58016us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.112168 (+1063906us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:58.228977 (+116809us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28152}
I20260926 01:55:58.193995 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8b2627427be143419dd9a44285379cd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:58.230597 9711 rpcz_store.cc:276] 0926 01:55:53.588848 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.588981 (+ 133us) service_pool.cc:224] Handling call
0926 01:55:57.225041 (+3636060us) ts_tablet_manager.cc:1151] Deleting tablet 1cba678b66744dcfb4d13ae07ebec33f
0926 01:55:57.265161 (+ 40120us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:57.854789 (+589628us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.989405 (+134616us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.100208 (+110803us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3635505}
W20260926 01:55:58.231766 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b2627427be143419dd9a44285379cd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:58.232435 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 161) took 4643 ms. Trace:
W20260926 01:55:58.107057 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 84003.5us
I20260926 01:55:58.232844 9711 rpcz_store.cc:276] 0926 01:55:53.589425 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.589567 (+ 142us) service_pool.cc:224] Handling call
0926 01:55:58.232058 (+4642491us) ts_tablet_manager.cc:1151] Deleting tablet 32e0fe60b8744dbb8ae2bafefe3e834d
0926 01:55:58.232124 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.232366 (+ 242us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4641874}
I20260926 01:55:58.233877 9711 tablet_replica.cc:333] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:58.242851 10030 consensus_queue.cc:1048] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:58.247730 9192 tablet_service.cc:1558] Processing DeleteTablet for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:57 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:58.247968 9858 consensus_queue.cc:1048] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.084s
I20260926 01:55:58.251741 9812 ts_tablet_manager.cc:1916] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:58.254633 9155 leader_election.cc:304] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:58.255307 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:58.256034 9155 leader_election.cc:304] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:58.257156 9989 raft_consensus.cc:2804] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:58.257522 9989 raft_consensus.cc:493] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:58.257850 9989 raft_consensus.cc:3060] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.266921 9805 raft_consensus.cc:2804] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:58.267302 9805 raft_consensus.cc:493] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:58.267592 9805 raft_consensus.cc:3060] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.271237 9195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42408 (request call id 214) took 1388 ms. Trace:
I20260926 01:55:58.271626 9195 rpcz_store.cc:276] 0926 01:55:56.882207 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.882319 (+ 112us) service_pool.cc:224] Handling call
0926 01:55:56.891732 (+ 9413us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:56.891820 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:56.949137 (+ 57317us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.130113 (+1180976us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:58.271051 (+140938us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:58.273105 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 226) took 1251 ms. Trace:
I20260926 01:55:58.273046 9190 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42408 (request call id 224) took 1266 ms. Trace:
I20260926 01:55:58.273459 9190 rpcz_store.cc:276] 0926 01:55:57.007098 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.007807 (+ 709us) service_pool.cc:224] Handling call
0926 01:55:57.009302 (+ 1495us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:57.009394 (+ 92us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:57.113567 (+104173us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.235765 (+1122198us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:58.272973 (+ 37208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:58.275332 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:58.275931 9070 raft_consensus.cc:3060] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.273439 9710 rpcz_store.cc:276] 0926 01:55:57.021336 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.062808 (+ 41472us) service_pool.cc:224] Handling call
0926 01:55:57.653908 (+591100us) ts_tablet_manager.cc:1151] Deleting tablet 3bf2fa5fbe4146b5adb27ff09de878d3
0926 01:55:57.653974 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:57.666304 (+ 12330us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:57.689064 (+ 22760us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.272973 (+583909us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76800,"tablet-delete.queue_time_us":117}
W20260926 01:55:58.284067 9154 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260926 01:55:58.289726 9341 tablet_bootstrap.cc:492] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:58.290321 9341 ts_tablet_manager.cc:1403] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.516s user 0.013s sys 0.003s
I20260926 01:55:58.112471 9339 raft_consensus.cc:359] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:58.162616 9006 leader_election.cc:343] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 23d2805b41d645d3bb905f26a380f383
W20260926 01:55:58.291254 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 128703us
I20260926 01:55:58.292619 9971 consensus_queue.cc:1048] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:55:58.295301 9006 consensus_peers.cc:597] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c27f1dd7bb0449a6a3fa371a8636e09b. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:58.295830 8537 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_0818779c2b1449eaaee8c359e22c3b86" } modify_external_catalogs: true
I20260926 01:55:58.295991 9006 leader_election.cc:304] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:58.296077 9070 raft_consensus.cc:2468] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:58.296705 10030 raft_consensus.cc:2804] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:58.296639 8537 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_0818779c2b1449eaaee8c359e22c3b86" } modify_external_catalogs: true
I20260926 01:55:58.297024 10030 raft_consensus.cc:493] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:58.297335 10030 raft_consensus.cc:3060] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.298856 8536 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_3478e66c0ff74bc4a27eb418e346e4e2" } new_table_name: "default.Table_7606d0ec00eb48549e0e381fdc608430" modify_external_catalogs: true
I20260926 01:55:58.303260 9711 raft_consensus.cc:2243] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:58.303802 9711 pending_rounds.cc:70] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Trying to abort 1 pending ops.
I20260926 01:55:58.304025 9711 pending_rounds.cc:77] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428234097324032 op_type: NO_OP noop_request { }
I20260926 01:55:58.304425 9711 raft_consensus.cc:2889] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:58.304745 9711 raft_consensus.cc:2272] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:55:58.308660 9006 leader_election.cc:343] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:58.309589 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 1cba678b66744dcfb4d13ae07ebec33f (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]) successfully deleted
W20260926 01:55:58.310385 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 32e0fe60b8744dbb8ae2bafefe3e834d because the tablet was not found. No further retry: Not found: Tablet not found: 32e0fe60b8744dbb8ae2bafefe3e834d
I20260926 01:55:58.310675 9339 raft_consensus.cc:385] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:58.311241 9339 raft_consensus.cc:740] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:58.327880 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 230) took 1210 ms. Trace:
I20260926 01:55:58.327935 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 3bf2fa5fbe4146b5adb27ff09de878d3 (table default.Table_ae4f6e78de7a4f6c8f96a3b179cf7c6d [id=f29287621a4a4cbea49066dde89fb252]) successfully deleted
I20260926 01:55:58.328301 9710 rpcz_store.cc:276] 0926 01:55:57.117796 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.117982 (+ 186us) service_pool.cc:224] Handling call
0926 01:55:58.327505 (+1209523us) ts_tablet_manager.cc:1151] Deleting tablet c36ea09cf15647c0a344c9ba9b289d61
0926 01:55:58.327603 (+ 98us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.327817 (+ 214us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":673447}
W20260926 01:55:58.329437 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet c36ea09cf15647c0a344c9ba9b289d61 because the tablet was not found. No further retry: Not found: Tablet not found: c36ea09cf15647c0a344c9ba9b289d61
I20260926 01:55:58.329844 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 229) took 1212 ms. Trace:
I20260926 01:55:58.330147 9710 rpcz_store.cc:276] 0926 01:55:57.117276 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.117434 (+ 158us) service_pool.cc:224] Handling call
0926 01:55:58.329565 (+1212131us) ts_tablet_manager.cc:1151] Deleting tablet 1cba678b66744dcfb4d13ae07ebec33f
0926 01:55:58.329614 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.329794 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":675246}
W20260926 01:55:58.331216 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 1cba678b66744dcfb4d13ae07ebec33f because the tablet was not found. No further retry: Not found: Tablet not found: 1cba678b66744dcfb4d13ae07ebec33f
I20260926 01:55:58.334757 9710 tablet_replica.cc:333] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
W20260926 01:55:58.335965 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 9bac55da5b164c0f981914d39c87c33e because the tablet was not found. No further retry: Not found: Tablet not found: 9bac55da5b164c0f981914d39c87c33e
W20260926 01:55:58.336627 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 70ef5265eb6244aa96c8195aa30cdfb0 because the tablet was not found. No further retry: Not found: Tablet not found: 70ef5265eb6244aa96c8195aa30cdfb0
W20260926 01:55:58.337216 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet c27f1dd7bb0449a6a3fa371a8636e09b because the tablet was not found. No further retry: Not found: Tablet not found: c27f1dd7bb0449a6a3fa371a8636e09b
W20260926 01:55:58.337757 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 39cd9130640d4b14beef42e9f1b36b99 because the tablet was not found. No further retry: Not found: Tablet not found: 39cd9130640d4b14beef42e9f1b36b99
I20260926 01:55:58.342887 9339 consensus_queue.cc:260] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.343676 10032 raft_consensus.cc:493] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:58.344193 10032 raft_consensus.cc:515] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.344647 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:58.315367 9341 raft_consensus.cc:359] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.352283 9805 raft_consensus.cc:515] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.349937 10032 leader_election.cc:290] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:58.379555 9070 raft_consensus.cc:2468] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:58.352874 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:58.346187 9341 raft_consensus.cc:385] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:58.380396 9341 raft_consensus.cc:740] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:58.380393 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_86c054848f05455c849277b21ead1bd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:58.382000 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:58.382156 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86c054848f05455c849277b21ead1bd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:58.384529 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:58.395991 9217 raft_consensus.cc:2468] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:58.400833 9155 leader_election.cc:304] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:58.403692 10045 raft_consensus.cc:2804] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:58.306246 10030 raft_consensus.cc:515] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.414860 10030 leader_election.cc:290] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:58.430171 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_5f5bcfd270554939a7accb824ebd31d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:58.431896 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f5bcfd270554939a7accb824ebd31d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:58.438516 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:58.440243 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:58.351624 9041 tablet_service.cc:1558] Processing DeleteTablet for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:57 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:58.361909 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:58.350224 9339 ts_tablet_manager.cc:1434] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.376s user 0.004s sys 0.002s
I20260926 01:55:58.446772 9072 raft_consensus.cc:3060] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:58.435819 9003 leader_election.cc:343] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:58.448230 9003 leader_election.cc:304] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:55:58.451670 9003 leader_election.cc:343] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:58.452287 9003 leader_election.cc:343] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 23d2805b41d645d3bb905f26a380f383
I20260926 01:55:58.452628 9003 leader_election.cc:304] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:55:58.453176 9003 raft_consensus.cc:2686] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 01:55:58.454052 10032 raft_consensus.cc:2749] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:58.400156 9989 raft_consensus.cc:515] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.460470 9340 tablet_bootstrap.cc:492] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:58.461007 9340 ts_tablet_manager.cc:1403] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.439s user 0.027s sys 0.021s
I20260926 01:55:58.463070 9072 raft_consensus.cc:2468] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:55:58.472946 10074 log.cc:927] Time spent T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260926 01:55:58.465903 9341 consensus_queue.cc:260] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.488140 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_4200bf9898c3412c9c196dda272aef75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:58.490120 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4200bf9898c3412c9c196dda272aef75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:58.495239 9003 leader_election.cc:343] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:58.501770 9003 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:58.514652 8952 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:55:58.517971 9340 raft_consensus.cc:359] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.518733 9340 raft_consensus.cc:385] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:58.521009 9340 raft_consensus.cc:740] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:58.522770 9340 consensus_queue.cc:260] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.534042 10061 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:55:58.539865 9339 tablet_bootstrap.cc:492] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:58.545748 9339 tablet_bootstrap.cc:654] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:58.547629 9711 ts_tablet_manager.cc:1916] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:58.553659 8577 master_runner.cc:187] Error getting master registration for 127.8.76.188:38143: OK, instance_id { permanent_uuid: "f75ba46482de422ea23daf17aa100849" instance_seqno: 1790387757206373 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:55:58.554672 8577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:55:58.565515 9340 ts_tablet_manager.cc:1434] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.104s user 0.006s sys 0.000s
I20260926 01:55:58.574832 9340 tablet_bootstrap.cc:492] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:58.575891 9812 ts_tablet_manager.cc:1929] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:58.589790 9812 log.cc:1199] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/75d4598596ff49378fc3536fff0ac1b1
I20260926 01:55:58.590027 9118 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:55:58.592059 9812 ts_tablet_manager.cc:1950] T 75d4598596ff49378fc3536fff0ac1b1 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:58.606133 10061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Bootstrap starting.
I20260926 01:55:58.608176 9340 tablet_bootstrap.cc:654] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:58.611886 9339 tablet_bootstrap.cc:492] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:58.612408 9339 ts_tablet_manager.cc:1403] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.011s
I20260926 01:55:58.627133 9989 leader_election.cc:290] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:58.627557 10066 raft_consensus.cc:493] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:58.637414 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 187) took 4195 ms. Trace:
I20260926 01:55:58.637933 9812 rpcz_store.cc:276] 0926 01:55:54.441836 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.445474 (+ 3638us) service_pool.cc:224] Handling call
0926 01:55:58.082011 (+3636537us) ts_tablet_manager.cc:1151] Deleting tablet 75d4598596ff49378fc3536fff0ac1b1
0926 01:55:58.082054 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.354111 (+272057us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.575729 (+221618us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.637339 (+ 61610us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3598339}
I20260926 01:55:58.639349 9339 raft_consensus.cc:359] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.639770 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 75d4598596ff49378fc3536fff0ac1b1 (table default.Table_8fa5adfde54e424d935ea9df94631389 [id=d706c478a4b54165a5e8943317180616]) successfully deleted
I20260926 01:55:58.640080 9339 raft_consensus.cc:385] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:58.640331 9339 raft_consensus.cc:740] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:58.641177 9339 consensus_queue.cc:260] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.642975 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 189) took 4200 ms. Trace:
I20260926 01:55:58.643347 9812 rpcz_store.cc:276] 0926 01:55:54.446265 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.446437 (+ 172us) service_pool.cc:224] Handling call
0926 01:55:58.642609 (+4196172us) ts_tablet_manager.cc:1151] Deleting tablet dfaca3fc02634f459083d23f3102f725
0926 01:55:58.642661 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.642910 (+ 249us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4158640}
W20260926 01:55:58.644749 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet dfaca3fc02634f459083d23f3102f725 because the tablet was not found. No further retry: Not found: Tablet not found: dfaca3fc02634f459083d23f3102f725
I20260926 01:55:58.645684 10048 raft_consensus.cc:493] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:58.646108 10048 raft_consensus.cc:515] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.648102 10048 leader_election.cc:290] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:58.649127 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 192) took 4035 ms. Trace:
I20260926 01:55:58.649538 9812 rpcz_store.cc:276] 0926 01:55:54.614224 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.614369 (+ 145us) service_pool.cc:224] Handling call
0926 01:55:58.648843 (+4034474us) ts_tablet_manager.cc:1151] Deleting tablet 23d2805b41d645d3bb905f26a380f383
0926 01:55:58.648889 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.649058 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4033817}
I20260926 01:55:58.651398 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:58.652222 9072 raft_consensus.cc:2393] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:58.654196 9812 tablet_replica.cc:333] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:58.655054 9812 raft_consensus.cc:2243] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:58.655869 9812 raft_consensus.cc:2272] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:58.657986 9812 ts_tablet_manager.cc:1916] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:58.672518 9340 tablet_bootstrap.cc:492] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:58.673149 9340 ts_tablet_manager.cc:1403] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.003s
I20260926 01:55:58.628048 10066 raft_consensus.cc:515] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:58.680680 10066 leader_election.cc:290] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:58.676177 9340 raft_consensus.cc:359] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.682228 9340 raft_consensus.cc:385] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:58.684651 9340 raft_consensus.cc:740] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:58.686944 9981 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:55:58.673722 9805 leader_election.cc:290] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:58.693307 10053 consensus_meta.cc:287] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.535s user 0.009s sys 0.003s
I20260926 01:55:58.693809 10053 raft_consensus.cc:515] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.695480 9211 raft_consensus.cc:2393] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:58.700685 9187 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42408 (request call id 227) took 1637 ms. Trace:
I20260926 01:55:58.701141 9187 rpcz_store.cc:276] 0926 01:55:57.063164 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.103307 (+ 40143us) service_pool.cc:224] Handling call
0926 01:55:58.004267 (+900960us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:58.004424 (+ 157us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:58.616793 (+612369us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.700083 (+ 83290us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:58.700616 (+ 533us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:55:58.690779 9340 consensus_queue.cc:260] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.707814 10053 leader_election.cc:290] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:55:58.716807 8536 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0558d37a3b5f4bcea43a8da7b99ded09 consensus state has no leader
I20260926 01:55:58.629225 10045 raft_consensus.cc:697] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:58.718689 10045 consensus_queue.cc:237] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.738785 9340 ts_tablet_manager.cc:1434] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.065s user 0.007s sys 0.001s
I20260926 01:55:58.752341 10048 raft_consensus.cc:493] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:58.752884 10048 raft_consensus.cc:515] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.753314 9340 tablet_bootstrap.cc:492] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:58.754916 10048 leader_election.cc:290] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:58.771749 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 355147us
I20260926 01:55:58.772848 9006 leader_election.cc:304] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:58.774636 10030 raft_consensus.cc:2804] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:58.775068 10030 raft_consensus.cc:493] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:58.775490 10030 raft_consensus.cc:3060] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:58.778649 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 375143us
I20260926 01:55:58.779696 9858 consensus_queue.cc:1048] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:55:58.782689 9155 leader_election.cc:304] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:58.787289 8539 master_service.cc:960] ReplaceTablet: received request to replace tablet 030c41a9718d47ddaa33382b20a60dab from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:58.797353 9339 ts_tablet_manager.cc:1434] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.185s user 0.007s sys 0.000s
I20260926 01:55:58.799548 9339 tablet_bootstrap.cc:492] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:58.801431 9340 tablet_bootstrap.cc:654] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:58.806442 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bac55da5b164c0f981914d39c87c33e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:58.807266 9072 raft_consensus.cc:2468] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:58.811436 9812 ts_tablet_manager.cc:1929] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:58.812633 9812 log.cc:1199] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/1cba678b66744dcfb4d13ae07ebec33f
W20260926 01:55:58.823691 8843 leader_election.cc:343] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 9bac55da5b164c0f981914d39c87c33e
I20260926 01:55:58.829694 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:58.831900 8900 raft_consensus.cc:3060] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.833526 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:58.835807 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f020590a8b428788e2bb516a11cc45" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:58.837486 9006 leader_election.cc:304] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:58.838621 10032 raft_consensus.cc:2749] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:55:58.840155 9812 ts_tablet_manager.cc:1950] T 1cba678b66744dcfb4d13ae07ebec33f P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:58.840267 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:58.841022 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 194) took 4222 ms. Trace:
I20260926 01:55:58.841346 9812 rpcz_store.cc:276] 0926 01:55:54.618611 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.618731 (+ 120us) service_pool.cc:224] Handling call
0926 01:55:58.654010 (+4035279us) ts_tablet_manager.cc:1151] Deleting tablet 1cba678b66744dcfb4d13ae07ebec33f
0926 01:55:58.654058 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.722193 (+ 68135us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.811271 (+ 89078us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:58.840958 (+ 29687us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4031962}
I20260926 01:55:58.842653 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 1cba678b66744dcfb4d13ae07ebec33f (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]) successfully deleted
I20260926 01:55:58.845403 8843 leader_election.cc:304] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
W20260926 01:55:58.845831 8646 debug-util.cc:398] Leaking SignalData structure 0x7b08000addc0 after lost signal to thread 8581
W20260926 01:55:58.847457 8646 debug-util.cc:398] Leaking SignalData structure 0x7b08000970e0 after lost signal to thread 8649
I20260926 01:55:58.851181 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 197) took 3906 ms. Trace:
I20260926 01:55:58.857820 9812 rpcz_store.cc:276] 0926 01:55:54.944317 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.944847 (+ 530us) service_pool.cc:224] Handling call
0926 01:55:58.850758 (+3905911us) ts_tablet_manager.cc:1151] Deleting tablet 17b8b0d54f5645e5963f1de217f94c84
0926 01:55:58.850803 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.851110 (+ 307us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3905285}
W20260926 01:55:58.858351 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 75757.5us
I20260926 01:55:58.785737 9341 ts_tablet_manager.cc:1434] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.495s user 0.005s sys 0.013s
I20260926 01:55:58.819144 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:58.859370 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 198) took 3914 ms. Trace:
I20260926 01:55:58.859685 9812 rpcz_store.cc:276] 0926 01:55:54.944707 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.945554 (+ 847us) service_pool.cc:224] Handling call
0926 01:55:58.858983 (+3913429us) ts_tablet_manager.cc:1151] Deleting tablet 0d068e93fd374a6091d7473db069a7fa
0926 01:55:58.859041 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.859309 (+ 268us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3912899}
I20260926 01:55:58.860688 9812 tablet_replica.cc:333] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:58.855006 9710 raft_consensus.cc:2243] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:58.854717 8900 raft_consensus.cc:2468] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:58.864880 10061 log.cc:826] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Log is configured to *not* fsync() on all Append() calls
W20260926 01:55:58.865053 9341 ts_tablet_manager.cc:1467] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532: Tablet startup took 574ms
I20260926 01:55:58.856053 9340 tablet_bootstrap.cc:492] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:58.857193 10048 raft_consensus.cc:2804] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:58.866417 10048 raft_consensus.cc:493] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:58.866794 10048 raft_consensus.cc:3060] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:55:58.865473 9341 ts_tablet_manager.cc:1469] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532: Trace:
0926 01:55:50.875374 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:50.877376 (+ 2002us) service_pool.cc:224] Handling call
0926 01:55:50.878685 (+ 1309us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:55:50.878770 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:55:50.971987 (+ 93217us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:51.052909 (+ 80922us) tablet_replica.cc:206] Creating consensus instance
0926 01:55:51.053331 (+ 422us) inbound_call.cc:177] Queueing success response
0926 01:55:57.774120 (+6720789us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0926 01:55:58.290792 (+516672us) ts_tablet_manager.cc:1435] Starting tablet replica
0926 01:55:58.313187 (+ 22395us) tablet_replica.cc:261] Starting instrumentation
0926 01:55:58.313799 (+ 612us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":144277,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":157,"tablet-open.queue_time_us":6720844,"thread_start_us":287618,"threads_started":1}
I20260926 01:55:58.870333 9341 tablet_bootstrap.cc:492] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:58.874830 9340 ts_tablet_manager.cc:1403] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.122s user 0.013s sys 0.001s
I20260926 01:55:58.877719 9340 raft_consensus.cc:359] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:58.878762 8504 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 125331us
W20260926 01:55:58.882329 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 17b8b0d54f5645e5963f1de217f94c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17b8b0d54f5645e5963f1de217f94c84 already in progress: creating tablet
W20260926 01:55:58.883188 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 0d068e93fd374a6091d7473db069a7fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d068e93fd374a6091d7473db069a7fa already in progress: creating tablet
I20260926 01:55:58.884245 9340 raft_consensus.cc:385] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:58.884618 9340 raft_consensus.cc:740] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:58.889761 9340 consensus_queue.cc:260] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.882637 8843 leader_election.cc:304] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:58.880317 10048 raft_consensus.cc:515] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.895459 9812 raft_consensus.cc:2243] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:58.897218 10048 leader_election.cc:290] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:55:58.879765 9003 consensus_peers.cc:597] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75d4598596ff49378fc3536fff0ac1b1. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:55:58.899183 10048 raft_consensus.cc:3060] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.901541 9339 tablet_bootstrap.cc:654] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:58.897307 9812 raft_consensus.cc:2272] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:58.919538 9340 ts_tablet_manager.cc:1434] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260926 01:55:58.920532 9812 ts_tablet_manager.cc:1916] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:58.920660 9340 tablet_bootstrap.cc:492] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:58.883339 9710 pending_rounds.cc:70] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Trying to abort 1 pending ops.
I20260926 01:55:58.920821 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_63324ad934044895b71a28a0e5286b8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:58.921468 9710 pending_rounds.cc:77] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333428260091604992 op_type: NO_OP noop_request { }
I20260926 01:55:58.921885 9710 raft_consensus.cc:2889] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260926 01:55:58.922212 9710 raft_consensus.cc:2272] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:58.883693 9989 raft_consensus.cc:2804] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:58.886356 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:58.922894 9211 raft_consensus.cc:2393] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
W20260926 01:55:58.923053 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63324ad934044895b71a28a0e5286b8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:58.923605 9989 raft_consensus.cc:493] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:58.923982 9989 raft_consensus.cc:3060] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.908057 10030 raft_consensus.cc:515] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.927520 10030 leader_election.cc:290] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:58.929342 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:58.931011 9003 leader_election.cc:343] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:58.934938 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 52396.1us
W20260926 01:55:58.946626 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 43968us
W20260926 01:55:58.948078 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0558d37a3b5f4bcea43a8da7b99ded09 consensus state has no leader
I20260926 01:55:58.959426 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bac55da5b164c0f981914d39c87c33e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:58.960021 9072 raft_consensus.cc:3060] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:58.961247 9339 tablet_bootstrap.cc:492] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:58.961942 9339 ts_tablet_manager.cc:1403] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.163s user 0.010s sys 0.002s
I20260926 01:55:58.964448 9155 leader_election.cc:304] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:58.965803 10045 raft_consensus.cc:2804] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:58.911649 10091 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:55:58.966933 10045 raft_consensus.cc:697] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:58.968151 10045 consensus_queue.cc:237] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:58.971172 9072 raft_consensus.cc:2468] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:55:58.973501 8843 leader_election.cc:304] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:55:58.976135 8843 raft_consensus.cc:2686] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260926 01:55:58.982975 9152 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:58.984216 9339 raft_consensus.cc:359] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:58.999243 9339 raft_consensus.cc:385] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.000478 9339 raft_consensus.cc:740] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.005642 9339 consensus_queue.cc:260] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:59.016463 9340 tablet_bootstrap.cc:654] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:59.031988 8843 leader_election.cc:343] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 9bac55da5b164c0f981914d39c87c33e
W20260926 01:55:59.032718 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 49784.4us
W20260926 01:55:59.033466 9152 leader_election.cc:343] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:55:59.034185 9152 leader_election.cc:343] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:59.035531 9339 ts_tablet_manager.cc:1434] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.073s user 0.007s sys 0.002s
I20260926 01:55:59.049350 9812 ts_tablet_manager.cc:1929] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:55:59.049938 9812 log.cc:1199] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/9bac55da5b164c0f981914d39c87c33e
I20260926 01:55:59.050812 9812 ts_tablet_manager.cc:1950] T 9bac55da5b164c0f981914d39c87c33e P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:59.051654 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 201) took 3633 ms. Trace:
I20260926 01:55:59.052014 9812 rpcz_store.cc:276] 0926 01:55:55.418362 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.418574 (+ 212us) service_pool.cc:224] Handling call
0926 01:55:58.860520 (+3441946us) ts_tablet_manager.cc:1151] Deleting tablet 9bac55da5b164c0f981914d39c87c33e
0926 01:55:58.860571 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.896936 (+ 36365us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b54001716d4. stack: 00007f94aec19323 00007f94aec19220 00007f94ae7bd574 0000000000516964 00000000005168f9 00007f94b5d16aa1 00007f94b5db5b22 00007f94b61f655c 00007f94b6e5fc3a 00007f94b6e90960 00007f94b6e76e6d 00007f94b6e76e19 00007f94b6e76da9 00007f94b6e76d71 00007f94b6e75f9d 00000000005bd4e4
0926 01:55:59.041456 (+144520us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.049213 (+ 7757us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.051584 (+ 2371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78173056,"tablet-delete.queue_time_us":3441380}
I20260926 01:55:59.053579 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 9bac55da5b164c0f981914d39c87c33e (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f]) successfully deleted
I20260926 01:55:59.054803 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 208) took 3305 ms. Trace:
I20260926 01:55:59.055143 9812 rpcz_store.cc:276] 0926 01:55:55.749215 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.753856 (+ 4641us) service_pool.cc:224] Handling call
0926 01:55:59.054405 (+3300549us) ts_tablet_manager.cc:1151] Deleting tablet 16f49a5a10ea4f7a821d89f18e3af560
0926 01:55:59.054447 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.054750 (+ 303us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3299778}
W20260926 01:55:59.056782 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 16f49a5a10ea4f7a821d89f18e3af560 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f49a5a10ea4f7a821d89f18e3af560 already in progress: creating tablet
I20260926 01:55:59.058487 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 209) took 2300 ms. Trace:
I20260926 01:55:59.059291 9812 rpcz_store.cc:276] 0926 01:55:56.757710 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.757871 (+ 161us) service_pool.cc:224] Handling call
0926 01:55:59.058036 (+2300165us) ts_tablet_manager.cc:1151] Deleting tablet 6d56f3a5dfed4ac78f1da92e22db558a
0926 01:55:59.058077 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.058403 (+ 326us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2299515}
W20260926 01:55:59.061046 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 6d56f3a5dfed4ac78f1da92e22db558a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d56f3a5dfed4ac78f1da92e22db558a already in progress: creating tablet
I20260926 01:55:59.062615 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 213) took 1673 ms. Trace:
I20260926 01:55:59.062958 9812 rpcz_store.cc:276] 0926 01:55:57.388788 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.388959 (+ 171us) service_pool.cc:224] Handling call
0926 01:55:59.062111 (+1673152us) ts_tablet_manager.cc:1151] Deleting tablet 2e12827ad93c47cf9013e238215ceb7a
0926 01:55:59.062176 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.062527 (+ 351us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672338}
W20260926 01:55:59.064256 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 2e12827ad93c47cf9013e238215ceb7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e12827ad93c47cf9013e238215ceb7a already in progress: creating tablet
I20260926 01:55:59.065182 9812 tablet_replica.cc:333] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:59.066071 9812 raft_consensus.cc:2243] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:55:59.067291 9812 raft_consensus.cc:2272] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:59.069507 9812 ts_tablet_manager.cc:1916] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:59.073028 9339 tablet_bootstrap.cc:492] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:59.085233 8540 master_service.cc:960] ReplaceTablet: received request to replace tablet 16f49a5a10ea4f7a821d89f18e3af560 from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:59.085391 9812 ts_tablet_manager.cc:1929] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:55:59.085997 9812 log.cc:1199] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/b659eca3c97144aaacd41e747aecd96d
I20260926 01:55:59.086928 9812 ts_tablet_manager.cc:1950] T b659eca3c97144aaacd41e747aecd96d P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:59.087716 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 216) took 1696 ms. Trace:
I20260926 01:55:59.088050 9812 rpcz_store.cc:276] 0926 01:55:57.390922 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.391033 (+ 111us) service_pool.cc:224] Handling call
0926 01:55:59.065029 (+1673996us) ts_tablet_manager.cc:1151] Deleting tablet b659eca3c97144aaacd41e747aecd96d
0926 01:55:59.065071 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.078076 (+ 13005us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.085244 (+ 7168us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.087663 (+ 2419us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673415}
I20260926 01:55:59.089469 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet b659eca3c97144aaacd41e747aecd96d (table default.Table_ee6cc37e9698474a8a3194f060b8c028 [id=0dae284524224264b7ccbd59cd975030]) successfully deleted
I20260926 01:55:59.090656 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 222) took 1554 ms. Trace:
I20260926 01:55:59.090999 9812 rpcz_store.cc:276] 0926 01:55:57.536359 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.536878 (+ 519us) service_pool.cc:224] Handling call
0926 01:55:59.090322 (+1553444us) ts_tablet_manager.cc:1151] Deleting tablet 55452e70a7fe48efbafdc7cbb29c6272
0926 01:55:59.090367 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.090601 (+ 234us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1515443}
W20260926 01:55:59.092449 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 55452e70a7fe48efbafdc7cbb29c6272 because the tablet was not found. No further retry: Not found: Tablet not found: 55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:55:59.093559 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 224) took 1554 ms. Trace:
I20260926 01:55:59.093971 9812 rpcz_store.cc:276] 0926 01:55:57.539135 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.539345 (+ 210us) service_pool.cc:224] Handling call
0926 01:55:59.093287 (+1553942us) ts_tablet_manager.cc:1151] Deleting tablet 03086ec5451743fdbf2d5726d43f31b1
0926 01:55:59.093330 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.093502 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513397}
I20260926 01:55:59.094893 9711 ts_tablet_manager.cc:1929] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:59.095548 9711 log.cc:1199] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/23d2805b41d645d3bb905f26a380f383
W20260926 01:55:59.096169 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 03086ec5451743fdbf2d5726d43f31b1 because the tablet was not found. No further retry: Not found: Tablet not found: 03086ec5451743fdbf2d5726d43f31b1
I20260926 01:55:59.096338 9711 ts_tablet_manager.cc:1950] T 23d2805b41d645d3bb905f26a380f383 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:55:59.097262 9812 tablet_replica.cc:333] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:59.098089 9812 raft_consensus.cc:2243] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:59.098724 9812 raft_consensus.cc:2272] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:59.100800 9812 ts_tablet_manager.cc:1916] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:59.017930 9989 raft_consensus.cc:515] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.109509 9341 tablet_bootstrap.cc:654] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.109861 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_87c331af29e7419593789c486dc50a13" } new_table_name: "default.Table_5f2783bf7cb9432f86a847b22c398ea4" modify_external_catalogs: true
I20260926 01:55:59.111500 10099 raft_consensus.cc:493] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:55:59.112190 10099 raft_consensus.cc:515] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.119998 9710 ts_tablet_manager.cc:1916] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:55:59.037326 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 72998.6us
I20260926 01:55:59.124300 9340 tablet_bootstrap.cc:492] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:59.124797 9340 ts_tablet_manager.cc:1403] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.204s user 0.012s sys 0.001s
I20260926 01:55:59.128108 9989 leader_election.cc:290] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:59.136559 9340 raft_consensus.cc:359] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.137249 9340 raft_consensus.cc:385] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.137507 9340 raft_consensus.cc:740] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.124989 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:59.138280 9072 raft_consensus.cc:3060] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.140924 10048 raft_consensus.cc:2749] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:59.127027 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:59.132650 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f020590a8b428788e2bb516a11cc45" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:59.143777 9073 raft_consensus.cc:2468] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:59.143911 9069 raft_consensus.cc:2393] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:59.134488 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:59.144815 9071 raft_consensus.cc:2393] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:59.137704 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:59.145735 9067 raft_consensus.cc:2393] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:59.147953 9072 raft_consensus.cc:2468] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:55:59.152808 9003 leader_election.cc:343] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: a601d850375c43679dde7a46cfe71d22
I20260926 01:55:59.155787 9339 tablet_bootstrap.cc:654] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.158564 10099 leader_election.cc:290] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:59.162294 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 160) took 5573 ms. Trace:
I20260926 01:55:59.162786 9711 rpcz_store.cc:276] 0926 01:55:53.589154 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.589619 (+ 465us) service_pool.cc:224] Handling call
0926 01:55:58.233688 (+4644069us) ts_tablet_manager.cc:1151] Deleting tablet 23d2805b41d645d3bb905f26a380f383
0926 01:55:58.233748 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.823346 (+589598us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.094733 (+271387us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.162219 (+ 67486us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4643487}
I20260926 01:55:59.164148 9711 tablet_replica.cc:333] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:55:59.165060 9711 raft_consensus.cc:2243] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
W20260926 01:55:59.165901 10012 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 9387
I20260926 01:55:59.166005 9711 raft_consensus.cc:2272] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:59.167605 9155 leader_election.cc:304] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:59.168473 9989 raft_consensus.cc:2804] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:59.168844 9989 raft_consensus.cc:493] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:59.169132 9989 raft_consensus.cc:3060] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.170814 10030 raft_consensus.cc:493] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:59.171257 10030 raft_consensus.cc:515] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.175043 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:55:59.176512 9003 leader_election.cc:343] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:59.182597 10030 leader_election.cc:290] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:59.185398 9981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42408 (request call id 220) took 2206 ms. Trace:
I20260926 01:55:59.185469 9340 consensus_queue.cc:260] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.185886 9981 rpcz_store.cc:276] 0926 01:55:56.978843 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.978952 (+ 109us) service_pool.cc:224] Handling call
0926 01:55:59.185298 (+2206346us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:57.455162 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:55:57.455641 (+ 479us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:55:57.455677 (+ 36us) alter_schema_op.cc:66] Acquired schema lock
0926 01:55:57.455759 (+ 82us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:55:57.455898 (+ 139us) alter_schema_op.cc:128] START. Timestamp: 1790387757455822
0926 01:55:57.455920 (+ 22us) op_driver.cc:348] REPLICATION: starting
0926 01:55:57.456552 (+ 632us) log.cc:844] Serialized 173 byte log entry
0926 01:55:58.628271 (+1171719us) op_driver.cc:464] REPLICATION: finished
0926 01:55:58.686680 (+ 58409us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:55:58.688464 (+ 1784us) log.cc:844] Serialized 5 byte log entry
0926 01:55:58.909797 (+221333us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.184294 (+274497us) log.cc:844] Serialized 14 byte log entry
0926 01:55:59.185140 (+ 846us) alter_schema_op.cc:73] Released schema lock
0926 01:55:59.185178 (+ 38us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38509,"prepare.queue_time_us":475368,"prepare.run_cpu_time_us":2210,"prepare.run_wall_time_us":96562,"raft.queue_time_us":108,"raft.run_cpu_time_us":984,"raft.run_wall_time_us":996,"replication_time_us":1172178,"spinlock_wait_cycles":66944,"thread_start_us":874270,"threads_started":4,"wal-append.queue_time_us":198626,"wal-append.run_cpu_time_us":1487,"wal-append.run_wall_time_us":1014441}]]}
I20260926 01:55:59.189298 9340 ts_tablet_manager.cc:1434] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.064s user 0.006s sys 0.001s
I20260926 01:55:59.190296 9340 tablet_bootstrap.cc:492] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:59.206398 9340 tablet_bootstrap.cc:654] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.210951 8874 tablet_service.cc:1511] Processing CreateTablet for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e (DEFAULT_TABLE table=default.Table_79241696f6954afba998044c7a981755 [id=1a74a8065133443cac2bfe40c8b5c56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.212208 8874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.225700 8535 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a]). Expected version 2 got 0
I20260926 01:55:59.229066 8535 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 4 got 3
I20260926 01:55:59.230163 8535 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:55:59.240150 10061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260926 01:55:59.241478 10061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Bootstrap complete.
I20260926 01:55:59.242103 9340 tablet_bootstrap.cc:492] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:59.242724 9340 ts_tablet_manager.cc:1403] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.005s
I20260926 01:55:59.252014 9041 tablet_service.cc:1511] Processing CreateTablet for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e (DEFAULT_TABLE table=default.Table_79241696f6954afba998044c7a981755 [id=1a74a8065133443cac2bfe40c8b5c56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.252476 8535 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:55:59.253252 9041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.253269 8535 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:55:59.195897 9187 tablet_service.cc:1511] Processing CreateTablet for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e (DEFAULT_TABLE table=default.Table_79241696f6954afba998044c7a981755 [id=1a74a8065133443cac2bfe40c8b5c56d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.259622 8875 tablet_service.cc:1511] Processing CreateTablet for tablet 504cca4587d64bc4964b298ba1a74fe9 (DEFAULT_TABLE table=default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.260764 8875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504cca4587d64bc4964b298ba1a74fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.190629 9155 leader_election.cc:304] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:59.261586 10082 raft_consensus.cc:2749] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260926 01:55:59.262625 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 72178.7us
I20260926 01:55:59.263078 10048 raft_consensus.cc:493] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:59.263373 9155 leader_election.cc:304] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:59.263520 10048 raft_consensus.cc:515] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.263689 9043 tablet_service.cc:1511] Processing CreateTablet for tablet 504cca4587d64bc4964b298ba1a74fe9 (DEFAULT_TABLE table=default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.264086 10082 raft_consensus.cc:2749] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:55:59.264834 9043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504cca4587d64bc4964b298ba1a74fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.265336 8877 tablet_service.cc:1511] Processing CreateTablet for tablet 3c7151ffe5d94a0599bbe3e6c8999371 (DEFAULT_TABLE table=default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.266410 8877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7151ffe5d94a0599bbe3e6c8999371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.270064 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:59.270787 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.270897 9211 raft_consensus.cc:2393] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:59.272470 10048 leader_election.cc:290] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:59.273737 9711 ts_tablet_manager.cc:1916] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:59.275460 9340 raft_consensus.cc:359] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.276098 9340 raft_consensus.cc:385] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.276353 9340 raft_consensus.cc:740] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.277002 9340 consensus_queue.cc:260] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.281560 10104 tablet.cc:1751] T 562604137d484e83a51de2f778d58fca P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:55:59.282757 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 3c7151ffe5d94a0599bbe3e6c8999371 (DEFAULT_TABLE table=default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.283864 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7151ffe5d94a0599bbe3e6c8999371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.288486 9044 tablet_service.cc:1511] Processing CreateTablet for tablet 3c7151ffe5d94a0599bbe3e6c8999371 (DEFAULT_TABLE table=default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.289695 9044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7151ffe5d94a0599bbe3e6c8999371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.254029 8535 catalog_manager.cc:5838] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:55:59.303802 8535 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
W20260926 01:55:59.305233 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28eaf7381f364a2682dd07f2bd90ede3 consensus state has no leader
I20260926 01:55:59.305787 8533 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 255) took 2214 ms. Trace:
I20260926 01:55:59.229355 9989 raft_consensus.cc:515] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:59.310336 9989 leader_election.cc:290] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:55:59.313853 9340 ts_tablet_manager.cc:1434] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.071s user 0.005s sys 0.001s
I20260926 01:55:59.315224 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:59.316049 9070 raft_consensus.cc:2393] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:59.318866 8843 leader_election.cc:304] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:59.326524 9340 tablet_bootstrap.cc:492] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:59.329073 10090 raft_consensus.cc:3060] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.332482 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5c66728bb54ec6beb4f0a2c25d9105 (DEFAULT_TABLE table=default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.333657 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5c66728bb54ec6beb4f0a2c25d9105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.335230 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5c66728bb54ec6beb4f0a2c25d9105 (DEFAULT_TABLE table=default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.336338 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5c66728bb54ec6beb4f0a2c25d9105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.306080 8533 rpcz_store.cc:276] 0926 01:55:57.090924 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.105452 (+ 14528us) service_pool.cc:224] Handling call
0926 01:55:57.106590 (+ 1138us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:57.107274 (+ 684us) catalog_manager.cc:3764] Apply alter schema
0926 01:55:57.107916 (+ 642us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:55:57.108139 (+ 223us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:55:58.094577 (+986438us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:55:59.305721 (+1211144us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:57.168128 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:57.168483 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:57.168506 (+ 23us) write_op.cc:435] Acquired schema lock
0926 01:55:57.168520 (+ 14us) tablet.cc:662] Decoding operations
0926 01:55:57.168747 (+ 227us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:57.168780 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:57.168799 (+ 19us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:57.168930 (+ 131us) tablet.cc:701] Row locks acquired
0926 01:55:57.168942 (+ 12us) write_op.cc:260] PREPARE: finished
0926 01:55:57.169046 (+ 104us) write_op.cc:270] Start()
0926 01:55:57.169118 (+ 72us) write_op.cc:276] Timestamp: P: 1790387757169000 usec, L: 0
0926 01:55:57.169131 (+ 13us) op_driver.cc:348] REPLICATION: starting
0926 01:55:57.169569 (+ 438us) log.cc:844] Serialized 405 byte log entry
0926 01:55:57.995898 (+826329us) op_driver.cc:464] REPLICATION: finished
0926 01:55:58.073578 (+ 77680us) write_op.cc:301] APPLY: starting
0926 01:55:58.073641 (+ 63us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:58.073744 (+ 103us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:58.073828 (+ 84us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:58.074003 (+ 175us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:58.074141 (+ 138us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:58.074182 (+ 41us) write_op.cc:312] APPLY: finished
0926 01:55:58.074399 (+ 217us) log.cc:844] Serialized 23 byte log entry
0926 01:55:58.075720 (+ 1321us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.075790 (+ 70us) write_op.cc:454] Released schema lock
0926 01:55:58.075821 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:58.094872 (+ 19051us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.094899 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":77496,"apply.run_cpu_time_us":1372,"apply.run_wall_time_us":21270,"mutex_wait_us":53407,"num_ops":1,"prepare.queue_time_us":58751,"prepare.run_cpu_time_us":1622,"prepare.run_wall_time_us":1622,"replication_time_us":826659}]]}
I20260926 01:55:59.035063 9152 leader_election.cc:304] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:55:59.257820 9190 tablet_service.cc:1511] Processing CreateTablet for tablet 504cca4587d64bc4964b298ba1a74fe9 (DEFAULT_TABLE table=default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:55:59.355909 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 320882us
I20260926 01:55:59.356926 10045 raft_consensus.cc:2804] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:59.357956 9006 leader_election.cc:304] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:59.358090 10045 raft_consensus.cc:697] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:59.358969 10099 raft_consensus.cc:2749] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:55:59.358950 10045 consensus_queue.cc:237] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:59.364149 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 98608.7us
I20260926 01:55:59.365018 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_93d21537b7c149098fb8fd3a5120fd27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:59.366803 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93d21537b7c149098fb8fd3a5120fd27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:59.368332 9339 tablet_bootstrap.cc:492] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:59.368822 9339 ts_tablet_manager.cc:1403] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.296s user 0.013s sys 0.004s
I20260926 01:55:59.370368 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:59.371600 9339 raft_consensus.cc:359] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.372260 9339 raft_consensus.cc:385] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.373335 9192 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5c66728bb54ec6beb4f0a2c25d9105 (DEFAULT_TABLE table=default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.374271 9339 raft_consensus.cc:740] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.375012 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 9db85241141f4a54a2b7dce15412a6f9 (DEFAULT_TABLE table=default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.374985 9339 consensus_queue.cc:260] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.376127 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db85241141f4a54a2b7dce15412a6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.376765 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 9db85241141f4a54a2b7dce15412a6f9 (DEFAULT_TABLE table=default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.377044 9339 ts_tablet_manager.cc:1434] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 01:55:59.378083 9339 tablet_bootstrap.cc:492] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:59.388993 9340 tablet_bootstrap.cc:654] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.389256 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5c66728bb54ec6beb4f0a2c25d9105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.390870 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504cca4587d64bc4964b298ba1a74fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.394186 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 030c41a9718d47ddaa33382b20a60dab
I20260926 01:55:59.394765 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db85241141f4a54a2b7dce15412a6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.395810 8532 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 257) took 2279 ms. Trace:
I20260926 01:55:59.395959 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_6e853298109b4319a558da622f72f21b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:59.397660 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e853298109b4319a558da622f72f21b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:59.400115 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 030c41a9718d47ddaa33382b20a60dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:57 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:55:59.400369 9045 tablet_service.cc:1558] Processing DeleteTablet for tablet 030c41a9718d47ddaa33382b20a60dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:57 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:59.403597 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet 030c41a9718d47ddaa33382b20a60dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:57 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:59.396144 8532 rpcz_store.cc:276] 0926 01:55:57.116122 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.952501 (+836379us) service_pool.cc:224] Handling call
0926 01:55:57.999072 (+ 46571us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607876 00007f717c1b6b34 00007f717c1b6ac2 00007f717c1b69f1 00007f717c1b696c 00007f717c1b5c22 00007f717a45c09c 00007f717a45b4d6
0926 01:55:58.000501 (+ 1429us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:58.001469 (+ 968us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:58.054716 (+ 53247us) catalog_manager.cc:2863] Locking tablets
0926 01:55:59.194121 (+1139405us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:59.393792 (+199671us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:59.394004 (+ 212us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:59.394092 (+ 88us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:59.395736 (+ 1644us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:59.312482 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:59.313046 (+ 564us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:59.313075 (+ 29us) write_op.cc:435] Acquired schema lock
0926 01:55:59.313092 (+ 17us) tablet.cc:662] Decoding operations
0926 01:55:59.313511 (+ 419us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:59.313558 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:59.313585 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:59.313911 (+ 326us) tablet.cc:701] Row locks acquired
0926 01:55:59.313924 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:55:59.314073 (+ 149us) write_op.cc:270] Start()
0926 01:55:59.314175 (+ 102us) write_op.cc:276] Timestamp: P: 1790387759313998 usec, L: 0
0926 01:55:59.314189 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:55:59.362950 (+ 48761us) log.cc:844] Serialized 728 byte log entry
0926 01:55:59.389876 (+ 26926us) op_driver.cc:464] REPLICATION: finished
0926 01:55:59.390298 (+ 422us) write_op.cc:301] APPLY: starting
0926 01:55:59.390389 (+ 91us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:59.390568 (+ 179us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:59.390692 (+ 124us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:59.391900 (+ 1208us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:59.392663 (+ 763us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:59.392720 (+ 57us) write_op.cc:312] APPLY: finished
0926 01:55:59.392933 (+ 213us) log.cc:844] Serialized 29 byte log entry
0926 01:55:59.393107 (+ 174us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:59.393185 (+ 78us) write_op.cc:454] Released schema lock
0926 01:55:59.393219 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:59.393454 (+ 235us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:59.393479 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":103788416,"child_traces":[["op",{"apply.queue_time_us":208,"apply.run_cpu_time_us":1687,"apply.run_wall_time_us":3139,"num_ops":2,"prepare.queue_time_us":103991,"prepare.run_cpu_time_us":2898,"prepare.run_wall_time_us":74433,"raft.queue_time_us":143,"raft.run_cpu_time_us":777,"raft.run_wall_time_us":778,"replication_time_us":75527}]]}
I20260926 01:55:59.406234 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_dff0511306454aa1a93fe774b3769823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:59.408830 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dff0511306454aa1a93fe774b3769823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:55:59.419564 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 52178us
I20260926 01:55:59.421883 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f020590a8b428788e2bb516a11cc45" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:55:59.426961 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f020590a8b428788e2bb516a11cc45" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:59.427649 9070 raft_consensus.cc:3060] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.438565 9710 ts_tablet_manager.cc:1929] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:59.439061 9710 log.cc:1199] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/72f666b1c069435b8ff0a3fae9f1e873
I20260926 01:55:59.440475 9710 ts_tablet_manager.cc:1950] T 72f666b1c069435b8ff0a3fae9f1e873 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:55:59.444327 9339 tablet_bootstrap.cc:654] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.450292 9341 tablet_bootstrap.cc:492] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:59.451683 9341 ts_tablet_manager.cc:1403] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.582s user 0.008s sys 0.011s
I20260926 01:55:59.454869 10082 raft_consensus.cc:493] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:59.455394 10082 raft_consensus.cc:515] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.456223 9341 raft_consensus.cc:359] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.456856 9341 raft_consensus.cc:385] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.457118 9341 raft_consensus.cc:740] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.457736 9341 consensus_queue.cc:260] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.458199 10082 leader_election.cc:290] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:59.459853 9341 ts_tablet_manager.cc:1434] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260926 01:55:59.460371 9340 tablet_bootstrap.cc:492] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:59.460860 9341 tablet_bootstrap.cc:492] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:55:59.460938 9340 ts_tablet_manager.cc:1403] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.008s
I20260926 01:55:59.465361 9340 raft_consensus.cc:359] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.468465 9044 tablet_service.cc:1511] Processing CreateTablet for tablet 9db85241141f4a54a2b7dce15412a6f9 (DEFAULT_TABLE table=default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:55:59.468802 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_dc2d02d1ff49427d81cc4c14d1f952c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:59.469779 9044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db85241141f4a54a2b7dce15412a6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:55:59.470005 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:59.470641 9072 raft_consensus.cc:2468] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:55:59.480604 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:55:59.481489 9211 raft_consensus.cc:2393] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:59.482641 9340 raft_consensus.cc:385] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.486173 9340 raft_consensus.cc:740] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:55:59.490753 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:59.491130 9340 consensus_queue.cc:260] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.537612 9340 ts_tablet_manager.cc:1434] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.076s user 0.006s sys 0.001s
I20260926 01:55:59.526711 9341 tablet_bootstrap.cc:654] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.539495 9340 tablet_bootstrap.cc:492] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:59.557036 9340 tablet_bootstrap.cc:654] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.558239 8532 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_a4d2c1ed01c74ff3a7260703edd509be" } new_table_name: "default.Table_4d73c5f209a7403bb51e1ef3ce6084bb" modify_external_catalogs: true
I20260926 01:55:59.561769 9339 tablet_bootstrap.cc:492] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:59.562367 9339 ts_tablet_manager.cc:1403] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.185s user 0.011s sys 0.008s
I20260926 01:55:59.567731 10090 raft_consensus.cc:2749] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:55:59.567832 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:59.568677 9212 raft_consensus.cc:2468] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:55:59.577759 9070 raft_consensus.cc:2468] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:55:59.558271 9155 leader_election.cc:304] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:59.583909 9989 raft_consensus.cc:2804] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:59.584409 9989 raft_consensus.cc:493] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:59.584830 9989 raft_consensus.cc:3060] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.566098 9339 raft_consensus.cc:359] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.593158 9339 raft_consensus.cc:385] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.593544 9339 raft_consensus.cc:740] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.597431 9340 tablet_bootstrap.cc:492] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:59.598145 9340 ts_tablet_manager.cc:1403] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.002s
I20260926 01:55:59.527132 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 243) took 1547 ms. Trace:
I20260926 01:55:59.598913 9710 rpcz_store.cc:276] 0926 01:55:57.979179 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.055542 (+ 76363us) service_pool.cc:224] Handling call
0926 01:55:58.334597 (+279055us) ts_tablet_manager.cc:1151] Deleting tablet 72f666b1c069435b8ff0a3fae9f1e873
0926 01:55:58.334636 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:58.862393 (+527757us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400111054. stack: 00007faf8230c323 00007faf8230c220 00007faf81eb0574 0000000000516964 00000000005168f9 00007faf89409aa1 00007faf894a8b22 00007faf898e955c 00007faf8a552c3a 00007faf8a583960 00007faf8a569e6d 00007faf8a569e19 00007faf8a569da9 00007faf8a569d71 00007faf8a568f9d 00000000005bd4e4
0926 01:55:59.219686 (+357293us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.418905 (+199219us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.527047 (+108142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":246863,"spinlock_wait_cycles":880002304,"tablet-delete.queue_time_us":216039}
I20260926 01:55:59.600946 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 72f666b1c069435b8ff0a3fae9f1e873 (table default.Table_87c331af29e7419593789c486dc50a13 [id=2fa9e93da5704e3b9970e16f5afe93bb]) successfully deleted
I20260926 01:55:59.601287 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 242) took 1622 ms. Trace:
I20260926 01:55:59.601606 9710 rpcz_store.cc:276] 0926 01:55:57.978855 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.055470 (+ 76615us) service_pool.cc:224] Handling call
0926 01:55:59.600988 (+1545518us) ts_tablet_manager.cc:1151] Deleting tablet 55452e70a7fe48efbafdc7cbb29c6272
0926 01:55:59.601034 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.601226 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1480457}
I20260926 01:55:59.602007 9339 consensus_queue.cc:260] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.607426 9339 ts_tablet_manager.cc:1434] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260926 01:55:59.613884 9340 raft_consensus.cc:359] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.617045 9339 tablet_bootstrap.cc:492] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:59.619292 8539 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_275ba09a70cd4eaca781dec90dc353b2" } modify_external_catalogs: true
I20260926 01:55:59.620329 9340 raft_consensus.cc:385] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.620267 8539 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_275ba09a70cd4eaca781dec90dc353b2" } modify_external_catalogs: true
I20260926 01:55:59.621513 9340 raft_consensus.cc:740] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.626806 9340 consensus_queue.cc:260] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.644429 9340 ts_tablet_manager.cc:1434] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260926 01:55:59.647367 9340 tablet_bootstrap.cc:492] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:59.649667 9339 tablet_bootstrap.cc:654] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.661620 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_4bbbc18bf03744a59245860ed3c2139e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:59.663441 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bbbc18bf03744a59245860ed3c2139e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:59.668861 9340 tablet_bootstrap.cc:654] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.685264 9339 tablet_bootstrap.cc:492] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:59.685829 9339 ts_tablet_manager.cc:1403] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.069s user 0.008s sys 0.003s
I20260926 01:55:59.689103 9339 raft_consensus.cc:359] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.671494 10061 raft_consensus.cc:359] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 4 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:55:59.705775 9339 raft_consensus.cc:385] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.708545 9339 raft_consensus.cc:740] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.711869 9339 consensus_queue.cc:260] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.714860 9340 tablet_bootstrap.cc:492] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:59.715453 9340 ts_tablet_manager.cc:1403] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.000s
I20260926 01:55:59.718075 9340 raft_consensus.cc:359] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.719908 9339 ts_tablet_manager.cc:1434] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260926 01:55:59.725970 9006 leader_election.cc:304] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:59.728794 10099 raft_consensus.cc:2749] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:55:59.730736 9339 tablet_bootstrap.cc:492] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:55:59.724906 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:55:59.732751 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:59.733747 9340 raft_consensus.cc:385] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 01:55:59.734486 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:59.742030 10090 raft_consensus.cc:493] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:59.742671 10090 raft_consensus.cc:515] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.734023 9340 raft_consensus.cc:740] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.747363 9340 consensus_queue.cc:260] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.749902 9340 ts_tablet_manager.cc:1434] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260926 01:55:59.755326 9340 tablet_bootstrap.cc:492] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:59.756147 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:55:59.760795 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:59.762387 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:55:59.763984 9070 raft_consensus.cc:2393] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:55:59.764050 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:55:59.770565 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 4 got 3
I20260926 01:55:59.757004 10090 leader_election.cc:290] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:59.771384 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:55:59.772287 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet ff004cf835c34095a836ad683cec9256
I20260926 01:55:59.773849 8537 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 263) took 1478 ms. Trace:
I20260926 01:55:59.777271 8877 tablet_service.cc:1558] Processing DeleteTablet for tablet ff004cf835c34095a836ad683cec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:58 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:55:59.778520 9044 tablet_service.cc:1558] Processing DeleteTablet for tablet ff004cf835c34095a836ad683cec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:58 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:55:59.778800 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_519e634e0d394eeba89b9eea66db9bd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:55:59.774129 8537 rpcz_store.cc:276] 0926 01:55:58.295271 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.295422 (+ 151us) service_pool.cc:224] Handling call
0926 01:55:58.297190 (+ 1768us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:58.393196 (+ 96006us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:55:58.393337 (+ 141us) catalog_manager.cc:2863] Locking tablets
0926 01:55:58.393883 (+ 546us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:55:58.584813 (+190930us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:55:58.585051 (+ 238us) catalog_manager.cc:2911] Committing in-memory state
0926 01:55:59.772180 (+1187129us) catalog_manager.cc:2915] Aborting table tasks
0926 01:55:59.773783 (+ 1603us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:58.475388 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:58.475890 (+ 502us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:58.475918 (+ 28us) write_op.cc:435] Acquired schema lock
0926 01:55:58.475930 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:58.509032 (+ 33102us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:58.509135 (+ 103us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:58.509172 (+ 37us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:55:58.509594 (+ 422us) tablet.cc:701] Row locks acquired
0926 01:55:58.509615 (+ 21us) write_op.cc:260] PREPARE: finished
0926 01:55:58.509783 (+ 168us) write_op.cc:270] Start()
0926 01:55:58.509895 (+ 112us) write_op.cc:276] Timestamp: P: 1790387758509732 usec, L: 0
0926 01:55:58.509918 (+ 23us) op_driver.cc:348] REPLICATION: starting
0926 01:55:58.510613 (+ 695us) log.cc:844] Serialized 728 byte log entry
0926 01:55:58.581021 (+ 70408us) op_driver.cc:464] REPLICATION: finished
0926 01:55:58.581444 (+ 423us) write_op.cc:301] APPLY: starting
0926 01:55:58.581504 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:58.581625 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:58.581746 (+ 121us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:58.582051 (+ 305us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:58.582211 (+ 160us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:55:58.582254 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:55:58.582513 (+ 259us) log.cc:844] Serialized 29 byte log entry
0926 01:55:58.582695 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.582785 (+ 90us) write_op.cc:454] Released schema lock
0926 01:55:58.582822 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:58.611746 (+ 28924us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.611782 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":250,"apply.run_cpu_time_us":1593,"apply.run_wall_time_us":1592,"num_ops":2,"prepare.queue_time_us":79293,"prepare.run_cpu_time_us":3123,"prepare.run_wall_time_us":136308,"raft.queue_time_us":3366,"raft.run_cpu_time_us":961,"raft.run_wall_time_us":962,"replication_time_us":70987}]]}
W20260926 01:55:59.780735 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_519e634e0d394eeba89b9eea66db9bd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:59.774264 9339 tablet_bootstrap.cc:654] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:59.733920 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 150266us
I20260926 01:55:59.736974 9341 tablet_bootstrap.cc:492] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:55:59.807435 9341 ts_tablet_manager.cc:1403] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.347s user 0.017s sys 0.000s
I20260926 01:55:59.758899 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff004cf835c34095a836ad683cec9256" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:55:59.811050 9340 tablet_bootstrap.cc:654] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:55:59.812781 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 4 got 3
I20260926 01:55:59.813933 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
W20260926 01:55:59.818009 9006 leader_election.cc:343] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: a601d850375c43679dde7a46cfe71d22
I20260926 01:55:59.818491 9006 leader_election.cc:304] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:55:59.819437 10030 raft_consensus.cc:2749] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:55:59.814934 8538 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
I20260926 01:55:59.836923 9340 tablet_bootstrap.cc:492] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:59.848564 8532 master_service.cc:960] ReplaceTablet: received request to replace tablet 030c41a9718d47ddaa33382b20a60dab from {username='slave'} at 127.0.0.1:42048
I20260926 01:55:59.850793 9340 ts_tablet_manager.cc:1403] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.096s user 0.013s sys 0.000s
I20260926 01:55:59.850907 9006 leader_election.cc:304] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:55:59.846710 9812 ts_tablet_manager.cc:1929] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:55:59.851917 9812 log.cc:1199] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/39cd9130640d4b14beef42e9f1b36b99
I20260926 01:55:59.851876 10030 raft_consensus.cc:2804] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:59.852247 10030 raft_consensus.cc:493] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:59.852576 10030 raft_consensus.cc:3060] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.857082 9812 ts_tablet_manager.cc:1950] T 39cd9130640d4b14beef42e9f1b36b99 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:55:59.857910 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 227) took 2316 ms. Trace:
I20260926 01:55:59.858220 9812 rpcz_store.cc:276] 0926 01:55:57.541323 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.544984 (+ 3661us) service_pool.cc:224] Handling call
0926 01:55:59.097093 (+1552109us) ts_tablet_manager.cc:1151] Deleting tablet 39cd9130640d4b14beef42e9f1b36b99
0926 01:55:59.097136 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.714701 (+617565us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.846566 (+131865us) tablet_metadata.cc:665] Metadata flushed
0926 01:55:59.857836 (+ 11270us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516691}
I20260926 01:55:59.859581 9812 tablet_replica.cc:333] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:55:59.859833 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 39cd9130640d4b14beef42e9f1b36b99 (table default.Table_92899e57b6c14f62b2a999bb2487f558 [id=45e5e00e818749a8beb091f6a62b7bf6]) successfully deleted
I20260926 01:55:59.860357 9812 raft_consensus.cc:2243] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:55:59.860450 9339 tablet_bootstrap.cc:492] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:55:59.860986 9339 ts_tablet_manager.cc:1403] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.135s user 0.009s sys 0.008s
I20260926 01:55:59.863303 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_d8e349dae18c4eb3ae3be35dc46bf47c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:55:59.865155 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8e349dae18c4eb3ae3be35dc46bf47c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:55:59.867935 9339 raft_consensus.cc:359] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.869928 9339 raft_consensus.cc:385] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.871145 9339 raft_consensus.cc:740] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.872576 9339 consensus_queue.cc:260] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.853892 9340 raft_consensus.cc:359] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:59.882988 9340 raft_consensus.cc:385] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.883370 9340 raft_consensus.cc:740] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.884127 9340 consensus_queue.cc:260] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:59.886425 9340 ts_tablet_manager.cc:1434] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20260926 01:55:59.887120 9812 raft_consensus.cc:2272] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:55:59.889489 8533 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d4b013d8752449f898dd39794a37a01f" } new_table_name: "default.Table_d2e4b69665074e9898068bafd9ed0bb9" modify_external_catalogs: true
I20260926 01:55:59.890179 9812 ts_tablet_manager.cc:1916] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:55:59.887807 9340 tablet_bootstrap.cc:492] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:55:59.897846 9340 tablet_bootstrap.cc:654] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:55:59.916332 9043 consensus_meta.cc:287] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.622s user 0.011s sys 0.000s
I20260926 01:55:59.931504 10061 raft_consensus.cc:740] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75ba46482de422ea23daf17aa100849, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.939513 10090 raft_consensus.cc:493] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:55:59.940037 10090 raft_consensus.cc:515] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.942165 10090 leader_election.cc:290] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:59.944351 9340 tablet_bootstrap.cc:492] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:55:59.944823 9340 ts_tablet_manager.cc:1403] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.005s
I20260926 01:55:59.951431 9340 raft_consensus.cc:359] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.934042 10129 raft_consensus.cc:493] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc00feec3f564608851f0112e97ab532)
I20260926 01:55:59.808698 9155 leader_election.cc:304] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:55:59.960404 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 28eaf7381f364a2682dd07f2bd90ede3 consensus state has no leader
W20260926 01:55:59.961326 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 55452e70a7fe48efbafdc7cbb29c6272 because the tablet was not found. No further retry: Not found: Tablet not found: 55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:55:59.962818 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 239) took 1985 ms. Trace:
I20260926 01:55:59.963188 9710 rpcz_store.cc:276] 0926 01:55:57.977636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.054712 (+ 77076us) service_pool.cc:224] Handling call
0926 01:55:59.806838 (+1752126us) ts_tablet_manager.cc:1151] Deleting tablet b659eca3c97144aaacd41e747aecd96d
0926 01:55:59.806891 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.962750 (+155859us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1667948}
W20260926 01:55:59.964608 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 155995us
I20260926 01:55:59.968976 9340 raft_consensus.cc:385] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.971309 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:55:59.971614 10030 raft_consensus.cc:515] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.971932 9070 raft_consensus.cc:2468] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:55:59.973590 10030 leader_election.cc:290] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:55:59.974984 9339 ts_tablet_manager.cc:1434] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.114s user 0.008s sys 0.005s
I20260926 01:55:59.972965 8843 leader_election.cc:304] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:55:59.977185 10090 raft_consensus.cc:2804] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:55:59.977560 10090 raft_consensus.cc:493] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:55:59.977856 10090 raft_consensus.cc:3060] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.970759 10129 raft_consensus.cc:515] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:55:59.981531 10129 leader_election.cc:290] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:59.975718 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:55:59.983201 9003 leader_election.cc:343] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:55:59.986008 10090 raft_consensus.cc:515] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:55:59.988384 9152 leader_election.cc:343] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:55:59.989641 10090 leader_election.cc:290] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:55:59.990250 9340 raft_consensus.cc:740] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.991240 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:55:59.986812 9341 raft_consensus.cc:359] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.991825 9070 raft_consensus.cc:3060] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:55:59.991916 9341 raft_consensus.cc:385] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:55:59.992766 9341 raft_consensus.cc:740] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:55:59.993534 9710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 244) took 1755 ms. Trace:
I20260926 01:55:59.993850 9710 rpcz_store.cc:276] 0926 01:55:58.237939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.247525 (+ 9586us) service_pool.cc:224] Handling call
0926 01:55:59.984140 (+1736615us) ts_tablet_manager.cc:1151] Deleting tablet b4c64a7619c4452f94b8d8b4f8bbbcf8
0926 01:55:59.984186 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.993466 (+ 9280us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1735907}
I20260926 01:55:59.995965 9341 consensus_queue.cc:260] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:55:59.983552 10082 raft_consensus.cc:2804] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:55:59.988767 9152 leader_election.cc:304] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:55:59.995302 9190 tablet_service.cc:1558] Processing DeleteTablet for tablet ff004cf835c34095a836ad683cec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:58 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:00.004338 10082 raft_consensus.cc:697] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:55:59.997207 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.006929 9211 raft_consensus.cc:2468] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:00.001893 9212 raft_consensus.cc:2393] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:55:59.982687 9989 raft_consensus.cc:515] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.006989 10082 consensus_queue.cc:237] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:00.008978 9710 tablet_replica.cc:333] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:00.014961 9710 raft_consensus.cc:2243] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:00.015507 9710 raft_consensus.cc:2272] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:56:00.016206 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet b659eca3c97144aaacd41e747aecd96d because the tablet was not found. No further retry: Not found: Tablet not found: b659eca3c97144aaacd41e747aecd96d
W20260926 01:56:00.017221 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 already in progress: creating tablet
I20260926 01:56:00.017885 9710 ts_tablet_manager.cc:1916] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:00.018124 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 030c41a9718d47ddaa33382b20a60dab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 030c41a9718d47ddaa33382b20a60dab already in progress: creating tablet
I20260926 01:56:00.019047 9340 consensus_queue.cc:260] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.019675 9989 leader_election.cc:290] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.019915 8900 raft_consensus.cc:3060] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.021015 8843 leader_election.cc:304] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:00.021165 9989 raft_consensus.cc:2749] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.022146 10090 raft_consensus.cc:3060] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.027452 8900 raft_consensus.cc:2468] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:00.029263 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:00.029830 9212 raft_consensus.cc:3060] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.031219 10061 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:00.033075 10090 raft_consensus.cc:2749] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.037560 9212 raft_consensus.cc:2468] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:00.044510 9340 ts_tablet_manager.cc:1434] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.096s user 0.006s sys 0.000s
I20260926 01:56:00.044541 9003 leader_election.cc:304] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:00.045523 9340 tablet_bootstrap.cc:492] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:00.045693 10132 raft_consensus.cc:2804] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:00.046269 10132 raft_consensus.cc:697] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:00.047153 10132 consensus_queue.cc:237] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.053264 8843 leader_election.cc:304] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:00.054406 10048 raft_consensus.cc:2804] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:00.054873 10048 raft_consensus.cc:697] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:00.048682 9070 raft_consensus.cc:2468] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:00.055648 10048 consensus_queue.cc:237] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.057482 10128 raft_consensus.cc:493] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.057973 10128 raft_consensus.cc:515] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.059736 10126 raft_consensus.cc:493] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.076217 9341 ts_tablet_manager.cc:1434] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.268s user 0.014s sys 0.000s
I20260926 01:56:00.090986 9341 tablet_bootstrap.cc:492] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:00.099103 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 133904us
W20260926 01:56:00.103034 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 114677us
W20260926 01:56:00.103765 9152 leader_election.cc:343] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:00.105166 9152 leader_election.cc:343] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:00.107466 10140 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:00.110890 10140 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:00.112764 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:00.113680 8900 raft_consensus.cc:2393] T 272bb87fe22c4720bcb3f0e07482dd8b P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:00.115337 9711 ts_tablet_manager.cc:1929] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:00.115759 9711 log.cc:1199] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/75d4598596ff49378fc3536fff0ac1b1
I20260926 01:56:00.116480 9711 ts_tablet_manager.cc:1950] T 75d4598596ff49378fc3536fff0ac1b1 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:00.124935 9339 tablet_bootstrap.cc:492] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:00.126881 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:00.136030 9341 tablet_bootstrap.cc:654] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.138687 10030 raft_consensus.cc:493] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc00feec3f564608851f0112e97ab532)
I20260926 01:56:00.139434 10030 raft_consensus.cc:515] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:00.141686 10030 leader_election.cc:290] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:00.151330 10061 sys_catalog.cc:565] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:56:00.161219 9812 ts_tablet_manager.cc:1929] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:00.161800 9812 log.cc:1199] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:56:00.161772 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:00.162585 9212 raft_consensus.cc:2393] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:56:00.163213 8952 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:56:00.166493 10137 consensus_queue.cc:1048] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:00.167289 10090 raft_consensus.cc:493] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.167968 10090 raft_consensus.cc:515] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.173123 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.175146 10090 leader_election.cc:290] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:00.177392 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:00.199452 9812 ts_tablet_manager.cc:1950] T 3bf2fa5fbe4146b5adb27ff09de878d3 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:00.200492 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 228) took 2654 ms. Trace:
I20260926 01:56:00.201239 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 162) took 6536 ms. Trace:
I20260926 01:56:00.200836 9812 rpcz_store.cc:276] 0926 01:55:57.545831 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.545995 (+ 164us) service_pool.cc:224] Handling call
0926 01:55:59.859393 (+2313398us) ts_tablet_manager.cc:1151] Deleting tablet 3bf2fa5fbe4146b5adb27ff09de878d3
0926 01:55:59.859440 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.926217 (+ 66777us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.161037 (+234820us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.200418 (+ 39381us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2278790}
I20260926 01:56:00.202349 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 3bf2fa5fbe4146b5adb27ff09de878d3 (table default.Table_ae4f6e78de7a4f6c8f96a3b179cf7c6d [id=f29287621a4a4cbea49066dde89fb252]) successfully deleted
I20260926 01:56:00.202754 9812 tablet_replica.cc:333] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:00.201704 9711 rpcz_store.cc:276] 0926 01:55:53.663822 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.663942 (+ 120us) service_pool.cc:224] Handling call
0926 01:55:59.163947 (+5500005us) ts_tablet_manager.cc:1151] Deleting tablet 75d4598596ff49378fc3536fff0ac1b1
0926 01:55:59.164009 (+ 62us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:55:59.835738 (+671729us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.115181 (+279443us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.200576 (+ 85395us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21605,"spinlock_wait_cycles":12160,"tablet-delete.queue_time_us":5499376}
I20260926 01:56:00.203554 9812 raft_consensus.cc:2243] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:00.204159 9812 raft_consensus.cc:2272] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:00.206121 9812 ts_tablet_manager.cc:1916] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:00.208259 8901 raft_consensus.cc:3060] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.209002 9341 tablet_bootstrap.cc:492] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:00.209506 9341 ts_tablet_manager.cc:1403] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.012s
I20260926 01:56:00.212244 9341 raft_consensus.cc:359] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.212906 9341 raft_consensus.cc:385] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.213178 9341 raft_consensus.cc:740] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:00.221438 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 75d4598596ff49378fc3536fff0ac1b1 (table default.Table_8fa5adfde54e424d935ea9df94631389 [id=d706c478a4b54165a5e8943317180616]) successfully deleted
I20260926 01:56:00.221587 9340 tablet_bootstrap.cc:654] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.223898 10128 leader_election.cc:290] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.225499 9339 tablet_bootstrap.cc:654] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.102090 10126 raft_consensus.cc:515] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.228726 9711 tablet_replica.cc:333] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:00.229763 9711 raft_consensus.cc:2243] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:00.214031 9341 consensus_queue.cc:260] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:56:00.241099 9006 leader_election.cc:343] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:56:00.241524 9006 leader_election.cc:304] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:00.241771 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]). Expected version 4 got 3
I20260926 01:56:00.243206 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:00.244082 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
I20260926 01:56:00.244851 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:00.245146 9341 ts_tablet_manager.cc:1434] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260926 01:56:00.246140 9341 tablet_bootstrap.cc:492] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:00.248261 9710 ts_tablet_manager.cc:1929] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:00.248667 9710 log.cc:1199] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/ff004cf835c34095a836ad683cec9256
I20260926 01:56:00.249339 9710 ts_tablet_manager.cc:1950] T ff004cf835c34095a836ad683cec9256 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:00.251523 9152 leader_election.cc:304] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:00.254679 10137 raft_consensus.cc:2749] T 272bb87fe22c4720bcb3f0e07482dd8b P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.257844 10126 leader_election.cc:290] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.258064 8539 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 2156ef6b24c04ea38181832e7f18ff49
I20260926 01:56:00.260142 9190 tablet_service.cc:1558] Processing DeleteTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:59 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:56:00.260975 9006 leader_election.cc:343] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 70ef5265eb6244aa96c8195aa30cdfb0
I20260926 01:56:00.261428 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:59 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:00.264564 9043 tablet_service.cc:1558] Processing DeleteTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:59 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:00.267355 10128 raft_consensus.cc:2749] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:00.274884 9711 raft_consensus.cc:2272] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:00.279838 9340 tablet_bootstrap.cc:492] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:00.280386 9340 ts_tablet_manager.cc:1403] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.235s user 0.005s sys 0.006s
I20260926 01:56:00.281855 9711 ts_tablet_manager.cc:1916] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:00.282637 8843 leader_election.cc:304] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:00.283458 10090 raft_consensus.cc:2749] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.287158 8902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943a990fa10948e8961c02e9999569f5" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:00.287796 8902 raft_consensus.cc:2468] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:00.288692 8898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:00.291620 9003 leader_election.cc:343] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 70ef5265eb6244aa96c8195aa30cdfb0
I20260926 01:56:00.292081 9003 leader_election.cc:304] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:00.293354 9003 leader_election.cc:304] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:00.294126 10143 raft_consensus.cc:2804] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:00.294514 10143 raft_consensus.cc:493] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:00.294850 10143 raft_consensus.cc:3060] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.296263 10130 raft_consensus.cc:493] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.297462 10130 raft_consensus.cc:515] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.293787 10127 raft_consensus.cc:2749] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:00.294149 9003 leader_election.cc:343] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:00.300231 9990 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 4 FOLLOWER]: Advancing to term 5
I20260926 01:56:00.301316 10130 leader_election.cc:290] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.303406 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b0e40e5e964b90bd2dd158bbc4497a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:00.309828 9003 leader_election.cc:343] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:00.311880 9990 pending_rounds.cc:85] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Aborting all ops after (but not including) 7
I20260926 01:56:00.306638 9340 raft_consensus.cc:359] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.312284 9990 pending_rounds.cc:107] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Aborting uncommitted WRITE_OP operation due to leader change: 4.8
I20260926 01:56:00.312628 9340 raft_consensus.cc:385] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.312906 9340 raft_consensus.cc:740] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:00.313246 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:00.314113 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d4b013d8752449f898dd39794a37a01f [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 2 got 0
I20260926 01:56:00.315052 8794 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 2156ef6b24c04ea38181832e7f18ff49
I20260926 01:56:00.314014 9990 pending_rounds.cc:107] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Aborting uncommitted WRITE_OP operation due to leader change: 4.9
I20260926 01:56:00.316859 9190 tablet_service.cc:1558] Processing DeleteTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:59 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:00.317798 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:59 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:00.322220 9190 tablet_service.cc:1511] Processing CreateTablet for tablet 6629d125861f4e27b61c10d585a35c10 (DEFAULT_TABLE table=default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.322685 9043 tablet_service.cc:1558] Processing DeleteTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:59 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:00.323499 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6629d125861f4e27b61c10d585a35c10. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:00.330912 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 79473.5us
I20260926 01:56:00.337905 9341 tablet_bootstrap.cc:654] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.335673 9340 consensus_queue.cc:260] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.341270 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:00.342574 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:00.342538 10143 raft_consensus.cc:515] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.344273 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
I20260926 01:56:00.344991 8537 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:00.350409 9812 ts_tablet_manager.cc:1929] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:00.350859 9812 log.cc:1199] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/c36ea09cf15647c0a344c9ba9b289d61
I20260926 01:56:00.351567 9812 ts_tablet_manager.cc:1950] T c36ea09cf15647c0a344c9ba9b289d61 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:00.352363 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 229) took 2805 ms. Trace:
I20260926 01:56:00.352661 9812 rpcz_store.cc:276] 0926 01:55:57.546798 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.547148 (+ 350us) service_pool.cc:224] Handling call
0926 01:56:00.202564 (+2655416us) ts_tablet_manager.cc:1151] Deleting tablet c36ea09cf15647c0a344c9ba9b289d61
0926 01:56:00.202613 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:00.342384 (+139771us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.350270 (+ 7886us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.352304 (+ 2034us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621751}
I20260926 01:56:00.354147 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet c36ea09cf15647c0a344c9ba9b289d61 (table default.Table_722ce830650d49ffa0b00028e056e1db [id=b10f8555970c4add9293d27073a1175b]) successfully deleted
I20260926 01:56:00.366333 10127 raft_consensus.cc:493] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.366822 10127 raft_consensus.cc:515] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.368991 10127 leader_election.cc:290] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.368896 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 6629d125861f4e27b61c10d585a35c10 (DEFAULT_TABLE table=default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.362097 10137 consensus_queue.cc:1048] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:00.370093 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6629d125861f4e27b61c10d585a35c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.372892 9043 tablet_service.cc:1511] Processing CreateTablet for tablet 6629d125861f4e27b61c10d585a35c10 (DEFAULT_TABLE table=default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.374121 9043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6629d125861f4e27b61c10d585a35c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.381546 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d4b013d8752449f898dd39794a37a01f [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 2 got 0
I20260926 01:56:00.382740 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:00.383509 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:00.384044 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:00.384747 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:00.385360 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
I20260926 01:56:00.385857 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]). Expected version 1 got 0
W20260926 01:56:00.387127 8540 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e76071c66b1d4635ac1a28110eb684fa consensus state has no leader
I20260926 01:56:00.387676 8540 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 271) took 1461 ms. Trace:
I20260926 01:56:00.387974 8540 rpcz_store.cc:276] 0926 01:55:58.926609 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.109519 (+182910us) service_pool.cc:224] Handling call
0926 01:55:59.111602 (+ 2083us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:55:59.112358 (+ 756us) catalog_manager.cc:3764] Apply alter schema
0926 01:55:59.112995 (+ 637us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:55:59.113226 (+ 231us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:55:59.309386 (+196160us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:56:00.387598 (+1078212us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:59.273943 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:59.274429 (+ 486us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:59.274478 (+ 49us) write_op.cc:435] Acquired schema lock
0926 01:55:59.274491 (+ 13us) tablet.cc:662] Decoding operations
0926 01:55:59.274812 (+ 321us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:59.274862 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:59.274889 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:59.287960 (+ 13071us) tablet.cc:701] Row locks acquired
0926 01:55:59.287981 (+ 21us) write_op.cc:260] PREPARE: finished
0926 01:55:59.288140 (+ 159us) write_op.cc:270] Start()
0926 01:55:59.288249 (+ 109us) write_op.cc:276] Timestamp: P: 1790387759288059 usec, L: 0
0926 01:55:59.288269 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:55:59.288901 (+ 632us) log.cc:844] Serialized 405 byte log entry
0926 01:55:59.303447 (+ 14546us) op_driver.cc:464] REPLICATION: finished
0926 01:55:59.303712 (+ 265us) write_op.cc:301] APPLY: starting
0926 01:55:59.303790 (+ 78us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:59.303899 (+ 109us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:59.304014 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:59.304208 (+ 194us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:59.304344 (+ 136us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:59.304388 (+ 44us) write_op.cc:312] APPLY: finished
0926 01:55:59.304619 (+ 231us) log.cc:844] Serialized 23 byte log entry
0926 01:55:59.304801 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:59.304874 (+ 73us) write_op.cc:454] Released schema lock
0926 01:55:59.304907 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:59.312156 (+ 7249us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:59.312186 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":1409,"apply.run_wall_time_us":1409,"num_ops":1,"prepare.queue_time_us":159453,"prepare.run_cpu_time_us":2558,"prepare.run_wall_time_us":38199,"raft.queue_time_us":5504,"raft.run_cpu_time_us":929,"raft.run_wall_time_us":929,"replication_time_us":14889}]]}
I20260926 01:56:00.393358 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_31abd87321d54403b8bbd41d2d7c8dc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:00.395522 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31abd87321d54403b8bbd41d2d7c8dc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:00.399843 10104 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:56:00.355934 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 230) took 2251 ms. Trace:
W20260926 01:56:00.405438 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 73729.8us
I20260926 01:56:00.398265 9340 ts_tablet_manager.cc:1434] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.118s user 0.007s sys 0.000s
I20260926 01:56:00.403483 10048 raft_consensus.cc:493] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.406432 10048 raft_consensus.cc:515] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.406970 9340 tablet_bootstrap.cc:492] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:00.408334 9265 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:56:00.408921 9265 heartbeater.cc:499] Master 127.8.76.188:38143 was elected leader, sending a full tablet report...
I20260926 01:56:00.408773 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1743e863cdd84f3f8ca413fa81cd1914" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:00.409382 8901 raft_consensus.cc:2468] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:00.414870 9003 leader_election.cc:304] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:00.415714 10127 raft_consensus.cc:2804] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:00.416080 10127 raft_consensus.cc:493] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:00.416385 10127 raft_consensus.cc:3060] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.420553 10048 leader_election.cc:290] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:00.401469 9812 rpcz_store.cc:276] 0926 01:55:58.104199 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.188162 (+ 83963us) service_pool.cc:224] Handling call
0926 01:56:00.355519 (+2167357us) ts_tablet_manager.cc:1151] Deleting tablet b4c64a7619c4452f94b8d8b4f8bbbcf8
0926 01:56:00.355564 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:00.355869 (+ 305us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2166457}
I20260926 01:56:00.424786 9339 tablet_bootstrap.cc:492] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:00.425243 9339 ts_tablet_manager.cc:1403] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.301s user 0.008s sys 0.009s
W20260926 01:56:00.425748 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4c64a7619c4452f94b8d8b4f8bbbcf8 already in progress: creating tablet
I20260926 01:56:00.426548 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:00.427119 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 236) took 1024 ms. Trace:
I20260926 01:56:00.427441 9812 rpcz_store.cc:276] 0926 01:55:59.403004 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.403440 (+ 436us) service_pool.cc:224] Handling call
0926 01:56:00.426738 (+1023298us) ts_tablet_manager.cc:1151] Deleting tablet 030c41a9718d47ddaa33382b20a60dab
0926 01:56:00.426787 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:00.427052 (+ 265us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022688}
W20260926 01:56:00.428813 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 030c41a9718d47ddaa33382b20a60dab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 030c41a9718d47ddaa33382b20a60dab already in progress: creating tablet
I20260926 01:56:00.454730 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.455430 9218 raft_consensus.cc:2393] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:56:00.455592 8877 tablet_service.cc:1511] Processing CreateTablet for tablet d0f4e81776f942ff96a9c1f274996c6a (DEFAULT_TABLE table=default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.456699 8877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f4e81776f942ff96a9c1f274996c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.458056 9067 raft_consensus.cc:2393] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:00.458214 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943a990fa10948e8961c02e9999569f5" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.458835 9211 raft_consensus.cc:2468] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:00.459323 10143 leader_election.cc:290] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.459302 9190 tablet_service.cc:1511] Processing CreateTablet for tablet d0f4e81776f942ff96a9c1f274996c6a (DEFAULT_TABLE table=default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.460379 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f4e81776f942ff96a9c1f274996c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.461774 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.464627 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet ff004cf835c34095a836ad683cec9256 (table default.Table_0818779c2b1449eaaee8c359e22c3b86 [id=5db31752ef1c4ff992bed8d186ba574e]) successfully deleted
W20260926 01:56:00.465405 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 2156ef6b24c04ea38181832e7f18ff49 because the tablet was not found. No further retry: Not found: Tablet not found: 2156ef6b24c04ea38181832e7f18ff49
I20260926 01:56:00.465930 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b0e40e5e964b90bd2dd158bbc4497a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:00.466091 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 2156ef6b24c04ea38181832e7f18ff49 because the tablet was not found. No further retry: Not found: Tablet not found: 2156ef6b24c04ea38181832e7f18ff49
I20260926 01:56:00.468304 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1743e863cdd84f3f8ca413fa81cd1914" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.468834 9212 raft_consensus.cc:2468] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:00.471788 9339 raft_consensus.cc:359] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:00.472448 9339 raft_consensus.cc:385] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.472693 9339 raft_consensus.cc:740] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:00.473294 9339 consensus_queue.cc:260] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:00.475029 8843 leader_election.cc:304] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:00.476137 10048 raft_consensus.cc:3060] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.422331 9340 tablet_bootstrap.cc:654] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.485106 8540 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_292982f73b37414ebefe3d3493c525c5" } modify_external_catalogs: true
I20260926 01:56:00.485714 8540 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_292982f73b37414ebefe3d3493c525c5" } modify_external_catalogs: true
I20260926 01:56:00.498854 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943a990fa10948e8961c02e9999569f5" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:00.499450 9212 raft_consensus.cc:3060] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.500998 8539 catalog_manager.cc:5838] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:56:00.505139 9044 tablet_service.cc:1511] Processing CreateTablet for tablet d0f4e81776f942ff96a9c1f274996c6a (DEFAULT_TABLE table=default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.506345 9044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f4e81776f942ff96a9c1f274996c6a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:00.508518 9006 leader_election.cc:343] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:00.509012 9006 leader_election.cc:304] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:00.512884 10145 raft_consensus.cc:2749] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:00.515527 9006 leader_election.cc:343] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:00.516090 9006 leader_election.cc:304] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:00.516788 10130 raft_consensus.cc:2749] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.426427 10127 raft_consensus.cc:515] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.519826 10127 leader_election.cc:290] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.524469 9192 tablet_service.cc:1511] Processing CreateTablet for tablet e85d17e9e48e4e54956e5c8460f7c9a8 (DEFAULT_TABLE table=default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.526269 10048 raft_consensus.cc:2749] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.526763 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e85d17e9e48e4e54956e5c8460f7c9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.526798 8880 tablet_service.cc:1511] Processing CreateTablet for tablet e85d17e9e48e4e54956e5c8460f7c9a8 (DEFAULT_TABLE table=default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.531162 9049 tablet_service.cc:1511] Processing CreateTablet for tablet e85d17e9e48e4e54956e5c8460f7c9a8 (DEFAULT_TABLE table=default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.532261 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e85d17e9e48e4e54956e5c8460f7c9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.535876 9341 tablet_bootstrap.cc:492] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:00.535867 9339 ts_tablet_manager.cc:1434] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.110s user 0.006s sys 0.001s
I20260926 01:56:00.536417 9341 ts_tablet_manager.cc:1403] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.291s user 0.014s sys 0.010s
I20260926 01:56:00.536854 9339 tablet_bootstrap.cc:492] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:00.551656 10048 raft_consensus.cc:493] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.552655 10048 raft_consensus.cc:515] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:00.555482 10048 leader_election.cc:290] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:00.556200 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f020590a8b428788e2bb516a11cc45" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.556439 10158 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:56:00.556632 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1f020590a8b428788e2bb516a11cc45" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:00.557435 9067 raft_consensus.cc:2393] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:56:00.558609 8843 leader_election.cc:304] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:00.562631 10048 raft_consensus.cc:3060] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.562963 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1743e863cdd84f3f8ca413fa81cd1914" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:00.563573 9211 raft_consensus.cc:3060] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.572234 9211 raft_consensus.cc:2468] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:00.574656 9341 raft_consensus.cc:359] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.575460 9341 raft_consensus.cc:385] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.575730 9341 raft_consensus.cc:740] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:00.576875 9006 leader_election.cc:304] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:00.578900 9212 raft_consensus.cc:2468] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:00.582685 10127 raft_consensus.cc:2804] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:00.587497 9340 tablet_bootstrap.cc:492] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:00.588020 9340 ts_tablet_manager.cc:1403] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.181s user 0.011s sys 0.000s
I20260926 01:56:00.596971 9341 consensus_queue.cc:260] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.502358 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d4b013d8752449f898dd39794a37a01f [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 2 got 0
I20260926 01:56:00.599251 9341 ts_tablet_manager.cc:1434] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20260926 01:56:00.600797 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_4d73c5f209a7403bb51e1ef3ce6084bb [id=2af856781c904c77a11ca9f4966ca44b]). Expected version 4 got 3
I20260926 01:56:00.601555 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:00.602414 9341 tablet_bootstrap.cc:492] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:00.603077 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1743e863cdd84f3f8ca413fa81cd1914" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:00.603785 8901 raft_consensus.cc:3060] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.603924 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "943a990fa10948e8961c02e9999569f5" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:00.604516 8900 raft_consensus.cc:3060] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.609438 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e85d17e9e48e4e54956e5c8460f7c9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.611044 9006 leader_election.cc:304] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:00.612886 10130 raft_consensus.cc:2804] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:00.613360 10130 raft_consensus.cc:697] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:00.614122 10130 consensus_queue.cc:237] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.622752 9340 raft_consensus.cc:359] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.623534 9340 raft_consensus.cc:385] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.623829 9340 raft_consensus.cc:740] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:00.624684 9340 consensus_queue.cc:260] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.626258 10134 consensus_queue.cc:1048] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:00.628270 9340 ts_tablet_manager.cc:1434] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20260926 01:56:00.628712 10135 consensus_queue.cc:1048] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:00.629264 9340 tablet_bootstrap.cc:492] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:00.631086 8901 raft_consensus.cc:2468] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:00.632622 9339 tablet_bootstrap.cc:654] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.660329 8900 raft_consensus.cc:2468] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:00.583465 10127 raft_consensus.cc:697] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:00.665491 10127 consensus_queue.cc:237] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.667785 9812 tablet_replica.cc:333] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:00.668861 9812 raft_consensus.cc:2243] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:00.669415 9812 raft_consensus.cc:2272] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:00.679687 9341 tablet_bootstrap.cc:654] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.680927 10090 raft_consensus.cc:493] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.681419 10090 raft_consensus.cc:515] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.683365 10090 leader_election.cc:290] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:00.700603 10134 raft_consensus.cc:493] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.701256 10134 raft_consensus.cc:515] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.703521 10134 leader_election.cc:290] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.690294 9192 tablet_service.cc:1511] Processing CreateTablet for tablet 094b7485380b46fe8e1fef23ac50b830 (DEFAULT_TABLE table=default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.709789 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:00.710501 9070 raft_consensus.cc:2468] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:56:00.711941 9155 leader_election.cc:304] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:00.712602 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 094b7485380b46fe8e1fef23ac50b830. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.713573 9812 ts_tablet_manager.cc:1916] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:00.713219 10137 raft_consensus.cc:2804] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:00.718791 10137 raft_consensus.cc:493] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:00.719203 10137 raft_consensus.cc:3060] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.722883 9711 ts_tablet_manager.cc:1929] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:00.723577 9711 log.cc:1199] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/39cd9130640d4b14beef42e9f1b36b99
I20260926 01:56:00.724515 8539 catalog_manager.cc:5838] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } }
I20260926 01:56:00.733657 10137 raft_consensus.cc:515] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.735373 9190 tablet_service.cc:1511] Processing CreateTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 (DEFAULT_TABLE table=default.Table_275ba09a70cd4eaca781dec90dc353b2 [id=3ec5a647a5fe475b8cdb6ff9ffe957f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.737356 10137 leader_election.cc:290] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:00.732916 9043 tablet_service.cc:1511] Processing CreateTablet for tablet 094b7485380b46fe8e1fef23ac50b830 (DEFAULT_TABLE table=default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.739692 9043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 094b7485380b46fe8e1fef23ac50b830. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.739810 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2156ef6b24c04ea38181832e7f18ff49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.747701 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:00.748474 8903 raft_consensus.cc:3060] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.756541 8903 raft_consensus.cc:2468] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:00.738142 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:00.757680 9070 raft_consensus.cc:3060] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:00.757961 9152 leader_election.cc:304] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
W20260926 01:56:00.761768 9003 consensus_peers.cc:597] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c36ea09cf15647c0a344c9ba9b289d61. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:00.762943 10166 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260926 01:56:00.766022 10137 raft_consensus.cc:2804] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:00.767503 10137 raft_consensus.cc:697] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:00.769327 9711 ts_tablet_manager.cc:1950] T 39cd9130640d4b14beef42e9f1b36b99 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:00.726383 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:00.771173 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:00.772009 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]). Expected version 2 got 0
I20260926 01:56:00.772797 8539 catalog_manager.cc:5838] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } }
I20260926 01:56:00.762092 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:00.775230 8900 raft_consensus.cc:2376] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1. Re-sending same reply.
I20260926 01:56:00.791347 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:00.792371 9339 tablet_bootstrap.cc:492] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:00.792815 9339 ts_tablet_manager.cc:1403] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.256s user 0.007s sys 0.008s
I20260926 01:56:00.795301 9049 tablet_service.cc:1511] Processing CreateTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 (DEFAULT_TABLE table=default.Table_275ba09a70cd4eaca781dec90dc353b2 [id=3ec5a647a5fe475b8cdb6ff9ffe957f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.795332 9198 tablet_service.cc:1511] Processing CreateTablet for tablet dd22a450f5914523871a088b24795e9f (DEFAULT_TABLE table=default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.795236 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9f54f648a55404fb6622b35d84368ee" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:00.796455 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd22a450f5914523871a088b24795e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.801224 8843 leader_election.cc:304] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:00.803627 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]). Expected version 1 got 0
I20260926 01:56:00.812515 9041 tablet_service.cc:1511] Processing CreateTablet for tablet dd22a450f5914523871a088b24795e9f (DEFAULT_TABLE table=default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.806699 8539 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:00.815332 8534 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_a82e6ba61ea64aaa8d10956665642633 [id=008c87e2e12f48519efdacc69466442a] alter complete (version 2)
I20260926 01:56:00.815397 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 094b7485380b46fe8e1fef23ac50b830 (DEFAULT_TABLE table=default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:56:00.816385 8533 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1f020590a8b428788e2bb516a11cc45 consensus state has no leader
I20260926 01:56:00.816578 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 094b7485380b46fe8e1fef23ac50b830. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.817765 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_bf5ded4035ed4595a340d90cab2ba64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:00.818831 10171 log.cc:927] Time spent T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20260926 01:56:00.820770 9341 tablet_bootstrap.cc:492] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:00.821269 9341 ts_tablet_manager.cc:1403] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.219s user 0.010s sys 0.004s
W20260926 01:56:00.822266 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf5ded4035ed4595a340d90cab2ba64c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:00.822724 10137 consensus_queue.cc:237] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.825860 8532 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b] alter complete (version 4)
I20260926 01:56:00.826565 8536 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9] alter complete (version 1)
I20260926 01:56:00.829908 9340 tablet_bootstrap.cc:654] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:00.832459 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 4efbc1e1ef35420db51742d99d5a32bf (DEFAULT_TABLE table=default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.851861 9339 raft_consensus.cc:359] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.852625 9339 raft_consensus.cc:385] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.852902 9339 raft_consensus.cc:740] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:00.853664 9339 consensus_queue.cc:260] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.855751 9339 ts_tablet_manager.cc:1434] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.063s user 0.009s sys 0.000s
I20260926 01:56:00.859843 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:52 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:00.860044 10130 raft_consensus.cc:493] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:00.860512 10130 raft_consensus.cc:515] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.862752 9045 tablet_service.cc:1558] Processing DeleteTablet for tablet 75d4598596ff49378fc3536fff0ac1b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:46 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:00.866875 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:00.871268 9006 leader_election.cc:343] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:00.873051 10130 leader_election.cc:290] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:00.870378 8536 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 44) took 2309 ms (client timeout 1000 ms). Trace:
I20260926 01:56:00.877130 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 163) took 7212 ms. Trace:
W20260926 01:56:00.873785 8536 rpcz_store.cc:269] 0926 01:55:58.560977 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.717588 (+156611us) service_pool.cc:224] Handling call
0926 01:55:58.764917 (+ 47329us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:00.870288 (+2105371us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:59.967079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:59.967578 (+ 499us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:59.967609 (+ 31us) write_op.cc:435] Acquired schema lock
0926 01:55:59.967625 (+ 16us) tablet.cc:662] Decoding operations
0926 01:55:59.967933 (+ 308us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:59.967979 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:59.968005 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:59.968182 (+ 177us) tablet.cc:701] Row locks acquired
0926 01:55:59.968199 (+ 17us) write_op.cc:260] PREPARE: finished
0926 01:55:59.968338 (+ 139us) write_op.cc:270] Start()
0926 01:55:59.968432 (+ 94us) write_op.cc:276] Timestamp: P: 1790387759968255 usec, L: 0
0926 01:55:59.968451 (+ 19us) op_driver.cc:348] REPLICATION: starting
0926 01:55:59.995556 (+ 27105us) log.cc:844] Serialized 356 byte log entry
0926 01:56:00.007382 (+ 11826us) op_driver.cc:464] REPLICATION: finished
0926 01:56:00.007815 (+ 433us) write_op.cc:301] APPLY: starting
0926 01:56:00.007890 (+ 75us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:00.007995 (+ 105us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:00.008113 (+ 118us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:00.008300 (+ 187us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:00.008441 (+ 141us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:00.008493 (+ 52us) write_op.cc:312] APPLY: finished
0926 01:56:00.008718 (+ 225us) log.cc:844] Serialized 23 byte log entry
0926 01:56:00.008885 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.008966 (+ 81us) write_op.cc:454] Released schema lock
0926 01:56:00.009009 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:00.030056 (+ 21047us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.030090 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9943,"reported_tablets":19,"spinlock_wait_cycles":82054528,"child_traces":[["op",{"apply.queue_time_us":281,"apply.run_cpu_time_us":1408,"apply.run_wall_time_us":1407,"num_ops":1,"prepare.queue_time_us":49376,"prepare.run_cpu_time_us":2634,"prepare.run_wall_time_us":62940,"raft.queue_time_us":340,"raft.run_cpu_time_us":862,"raft.run_wall_time_us":865,"replication_time_us":38814}]]}
I20260926 01:56:00.880363 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_bf92ccd2dbe1453d943f371ffec0badb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:56:00.878178 9711 rpcz_store.cc:276] 0926 01:55:53.664513 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.664638 (+ 125us) service_pool.cc:224] Handling call
0926 01:56:00.222377 (+6557739us) ts_tablet_manager.cc:1151] Deleting tablet 39cd9130640d4b14beef42e9f1b36b99
0926 01:56:00.228592 (+ 6215us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:00.365969 (+137377us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.722707 (+356738us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:00.877055 (+154348us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6557094}
W20260926 01:56:00.882047 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf92ccd2dbe1453d943f371ffec0badb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:00.882920 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 39cd9130640d4b14beef42e9f1b36b99 (table default.Table_92899e57b6c14f62b2a999bb2487f558 [id=45e5e00e818749a8beb091f6a62b7bf6]) successfully deleted
I20260926 01:56:00.883996 9711 tablet_replica.cc:333] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:00.883874 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:00.896472 9340 tablet_bootstrap.cc:492] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:00.897064 9340 ts_tablet_manager.cc:1403] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.268s user 0.007s sys 0.005s
I20260926 01:56:00.897518 9041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd22a450f5914523871a088b24795e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.926646 9341 raft_consensus.cc:359] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.927337 9341 raft_consensus.cc:385] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.927577 9341 raft_consensus.cc:740] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:00.931324 9049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2156ef6b24c04ea38181832e7f18ff49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.898725 9339 tablet_bootstrap.cc:492] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:00.802758 8882 tablet_service.cc:1511] Processing CreateTablet for tablet 2156ef6b24c04ea38181832e7f18ff49 (DEFAULT_TABLE table=default.Table_275ba09a70cd4eaca781dec90dc353b2 [id=3ec5a647a5fe475b8cdb6ff9ffe957f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.941061 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2156ef6b24c04ea38181832e7f18ff49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.947839 10048 raft_consensus.cc:2749] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.957121 10137 consensus_queue.cc:1048] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:00.966818 10127 consensus_queue.cc:1048] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:00.970664 9198 tablet_service.cc:1511] Processing CreateTablet for tablet 5f300124bc3a4177a606b116bfef3a65 (DEFAULT_TABLE table=default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.973455 9045 tablet_service.cc:1511] Processing CreateTablet for tablet 5f300124bc3a4177a606b116bfef3a65 (DEFAULT_TABLE table=default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.974695 9045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f300124bc3a4177a606b116bfef3a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:00.974851 10090 raft_consensus.cc:2749] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:00.977941 9340 raft_consensus.cc:359] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.978690 9340 raft_consensus.cc:385] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:00.978941 9340 raft_consensus.cc:740] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:56:00.979807 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0558d37a3b5f4bcea43a8da7b99ded09 consensus state has no leader
I20260926 01:56:00.979552 9340 consensus_queue.cc:260] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:00.981568 9340 ts_tablet_manager.cc:1434] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.084s user 0.005s sys 0.000s
W20260926 01:56:00.986860 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 185680us
I20260926 01:56:00.987401 9043 tablet_service.cc:1511] Processing CreateTablet for tablet 4efbc1e1ef35420db51742d99d5a32bf (DEFAULT_TABLE table=default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:00.991577 8880 tablet_service.cc:1511] Processing CreateTablet for tablet dd22a450f5914523871a088b24795e9f (DEFAULT_TABLE table=default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.006628 9067 raft_consensus.cc:1275] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Refusing update from remote peer bc00feec3f564608851f0112e97ab532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:56:01.008433 10137 consensus_queue.cc:1048] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:01.009423 9265 heartbeater.cc:499] Master 127.8.76.188:38143 was elected leader, sending a full tablet report...
I20260926 01:56:01.021533 9340 tablet_bootstrap.cc:492] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:01.023049 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd22a450f5914523871a088b24795e9f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:01.026554 9044 consensus_meta.cc:287] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.509s user 0.013s sys 0.005s
I20260926 01:56:01.027627 9340 tablet_bootstrap.cc:654] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:00.919117 9003 leader_election.cc:343] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:01.033043 9003 leader_election.cc:304] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:56:01.035396 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 116275us
I20260926 01:56:01.035324 9341 consensus_queue.cc:260] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:01.037250 9187 tablet_service.cc:1511] Processing CreateTablet for tablet cdd0f4921da4436aa259486227a180d5 (DEFAULT_TABLE table=default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.044656 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efbc1e1ef35420db51742d99d5a32bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.044773 8540 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 3571967749844fd7859cf179a93cf403
I20260926 01:56:01.046046 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f300124bc3a4177a606b116bfef3a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.048287 8540 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 285) took 1063 ms. Trace:
I20260926 01:56:01.049198 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 3571967749844fd7859cf179a93cf403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:00 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.049631 9341 ts_tablet_manager.cc:1434] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.228s user 0.008s sys 0.000s
I20260926 01:56:01.050747 9341 tablet_bootstrap.cc:492] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:01.049769 8540 rpcz_store.cc:276] 0926 01:55:59.984511 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.484772 (+500261us) service_pool.cc:224] Handling call
0926 01:56:00.486097 (+ 1325us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:56:00.546513 (+ 60416us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:56:00.546718 (+ 205us) catalog_manager.cc:2863] Locking tablets
0926 01:56:00.979879 (+433161us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:56:01.005700 (+ 25821us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:56:01.044424 (+ 38724us) catalog_manager.cc:2911] Committing in-memory state
0926 01:56:01.044661 (+ 237us) catalog_manager.cc:2915] Aborting table tasks
0926 01:56:01.048216 (+ 3555us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:00.982122 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:00.983552 (+ 1430us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:00.983584 (+ 32us) write_op.cc:435] Acquired schema lock
0926 01:56:00.983603 (+ 19us) tablet.cc:662] Decoding operations
0926 01:56:00.984948 (+ 1345us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:00.985000 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:00.985025 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:00.985353 (+ 328us) tablet.cc:701] Row locks acquired
0926 01:56:00.985366 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:56:00.985514 (+ 148us) write_op.cc:270] Start()
0926 01:56:00.985620 (+ 106us) write_op.cc:276] Timestamp: P: 1790387760985433 usec, L: 0
0926 01:56:00.985634 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:56:00.986264 (+ 630us) log.cc:844] Serialized 793 byte log entry
0926 01:56:01.003801 (+ 17537us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.004241 (+ 440us) write_op.cc:301] APPLY: starting
0926 01:56:01.004304 (+ 63us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:01.004428 (+ 124us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:01.004522 (+ 94us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:01.004817 (+ 295us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:01.004955 (+ 138us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:56:01.004993 (+ 38us) write_op.cc:312] APPLY: finished
0926 01:56:01.005202 (+ 209us) log.cc:844] Serialized 29 byte log entry
0926 01:56:01.005367 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.005449 (+ 82us) write_op.cc:454] Released schema lock
0926 01:56:01.005484 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:01.030614 (+ 25130us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.030642 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":269,"apply.run_cpu_time_us":1497,"apply.run_wall_time_us":1657,"num_ops":2,"prepare.queue_time_us":140,"prepare.run_cpu_time_us":3758,"prepare.run_wall_time_us":48354,"raft.queue_time_us":280,"raft.run_cpu_time_us":768,"raft.run_wall_time_us":768,"replication_time_us":18033}]]}
I20260926 01:56:01.056311 9052 tablet_service.cc:1511] Processing CreateTablet for tablet cdd0f4921da4436aa259486227a180d5 (DEFAULT_TABLE table=default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.057379 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 3571967749844fd7859cf179a93cf403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:00 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:01.049412 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_0a43386150b14b53ad954d8f63768297"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:01.060956 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a43386150b14b53ad954d8f63768297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:01.057529 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd0f4921da4436aa259486227a180d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.062599 9184 tablet_service.cc:1511] Processing CreateTablet for tablet 812fe778734d4210b744ed8e45d412f1 (DEFAULT_TABLE table=default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.065362 8540 master_service.cc:960] ReplaceTablet: received request to replace tablet cbf8818e65d74a32ba21dcb8a9a56a62 from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:01.073217 9184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812fe778734d4210b744ed8e45d412f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.058208 10130 raft_consensus.cc:2749] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:01.087160 9043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efbc1e1ef35420db51742d99d5a32bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.095316 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd0f4921da4436aa259486227a180d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.098382 10145 consensus_queue.cc:1048] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:01.103089 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet 3571967749844fd7859cf179a93cf403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:00 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.103439 8874 tablet_service.cc:1511] Processing CreateTablet for tablet cdd0f4921da4436aa259486227a180d5 (DEFAULT_TABLE table=default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.103804 8877 tablet_service.cc:1511] Processing CreateTablet for tablet 4efbc1e1ef35420db51742d99d5a32bf (DEFAULT_TABLE table=default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.104744 8874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd0f4921da4436aa259486227a180d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.105464 8875 tablet_service.cc:1511] Processing CreateTablet for tablet 5f300124bc3a4177a606b116bfef3a65 (DEFAULT_TABLE table=default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.106791 8877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efbc1e1ef35420db51742d99d5a32bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.107710 8875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f300124bc3a4177a606b116bfef3a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.135078 9070 raft_consensus.cc:2468] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:56:01.136897 8537 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 45) took 1319 ms (client timeout 1000 ms). Trace:
I20260926 01:56:01.139859 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 943a990fa10948e8961c02e9999569f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:51 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:01.140296 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:51458
W20260926 01:56:01.142757 8535 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 34) took 3182 ms (client timeout 1000 ms). Trace:
W20260926 01:56:01.148345 8532 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 46) took 1204 ms (client timeout 1000 ms). Trace:
I20260926 01:56:01.150604 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 812fe778734d4210b744ed8e45d412f1 (DEFAULT_TABLE table=default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.154627 10130 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:01.155232 10130 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:01.157315 10130 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:01.158938 9339 tablet_bootstrap.cc:654] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:01.161367 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17b8b0d54f5645e5963f1de217f94c84" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:01.163318 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.163615 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.164592 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.165390 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:52 UTC) from {username='slave'} at 127.0.0.1:58226
W20260926 01:56:01.165534 8531 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 45) took 2247 ms (client timeout 1000 ms). Trace:
I20260926 01:56:01.166146 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 943a990fa10948e8961c02e9999569f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:51 UTC) from {username='slave'} at 127.0.0.1:58226
W20260926 01:56:01.166580 8531 rpcz_store.cc:269] 0926 01:55:58.918528 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.008208 (+ 89680us) service_pool.cc:224] Handling call
0926 01:55:59.023430 (+ 15222us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:01.165449 (+2142019us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36298,"reported_tablets":16,"spinlock_wait_cycles":2819968}
I20260926 01:56:01.167456 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 812fe778734d4210b744ed8e45d412f1 (DEFAULT_TABLE table=default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.168321 8531 master_service.cc:960] ReplaceTablet: received request to replace tablet 1f6d9fedc7b241b59b25d591c7dc2fea from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:01.168723 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812fe778734d4210b744ed8e45d412f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:01.177857 9003 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:01.182873 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812fe778734d4210b744ed8e45d412f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.191474 8875 tablet_service.cc:1558] Processing DeleteTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.191171 8877 tablet_service.cc:1558] Processing DeleteTablet for tablet b659eca3c97144aaacd41e747aecd96d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:55 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.196836 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet 39cd9130640d4b14beef42e9f1b36b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:49 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.211479 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cba678b66744dcfb4d13ae07ebec33f with delete_type TABLET_DATA_DELETED (replaced by tablet 880d77d42c394e6a9cf0b3768a915402) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.218411 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 with delete_type TABLET_DATA_DELETED (replaced by tablet b612e8955d5f42dd8658320031fe36ca) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.219561 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bac55da5b164c0f981914d39c87c33e with delete_type TABLET_DATA_DELETED (replaced by tablet 0558d37a3b5f4bcea43a8da7b99ded09) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.220960 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (DEFAULT_TABLE table=default.Table_86c054848f05455c849277b21ead1bd7 [id=7eccc1870b21429ea89fa66914b317dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.222016 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:01.162227 8534 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 43) took 4402 ms (client timeout 1000 ms). Trace:
W20260926 01:56:01.169967 8538 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 44) took 3291 ms (client timeout 1000 ms). Trace:
W20260926 01:56:01.226724 8538 rpcz_store.cc:269] 0926 01:55:57.879064 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.910807 (+ 31743us) service_pool.cc:224] Handling call
0926 01:55:58.016642 (+105835us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:01.169896 (+3153254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41245,"reported_tablets":16,"spinlock_wait_cycles":236617472}
I20260926 01:56:01.228302 8538 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_e3f193c692be4f919e17e315a529b202" } modify_external_catalogs: true
I20260926 01:56:01.228929 8538 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_e3f193c692be4f919e17e315a529b202" } modify_external_catalogs: true
W20260926 01:56:01.225385 8534 rpcz_store.cc:269] 0926 01:55:56.766634 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.863164 (+ 96530us) service_pool.cc:224] Handling call
0926 01:55:56.925883 (+ 62719us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:01.162092 (+4236209us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:58.239796 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:58.240308 (+ 512us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:58.240342 (+ 34us) write_op.cc:435] Acquired schema lock
0926 01:55:58.240361 (+ 19us) tablet.cc:662] Decoding operations
0926 01:55:58.240701 (+ 340us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:58.240747 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:58.240776 (+ 29us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:58.240952 (+ 176us) tablet.cc:701] Row locks acquired
0926 01:55:58.240972 (+ 20us) write_op.cc:260] PREPARE: finished
0926 01:55:58.241127 (+ 155us) write_op.cc:270] Start()
0926 01:55:58.241223 (+ 96us) write_op.cc:276] Timestamp: P: 1790387758241064 usec, L: 0
0926 01:55:58.241242 (+ 19us) op_driver.cc:348] REPLICATION: starting
0926 01:55:58.241844 (+ 602us) log.cc:844] Serialized 356 byte log entry
0926 01:55:58.268040 (+ 26196us) op_driver.cc:464] REPLICATION: finished
0926 01:55:58.283356 (+ 15316us) write_op.cc:301] APPLY: starting
0926 01:55:58.283429 (+ 73us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:58.283534 (+ 105us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:58.283658 (+ 124us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:58.283890 (+ 232us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:58.284043 (+ 153us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:58.284085 (+ 42us) write_op.cc:312] APPLY: finished
0926 01:55:58.284301 (+ 216us) log.cc:844] Serialized 23 byte log entry
0926 01:55:58.284478 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.284566 (+ 88us) write_op.cc:454] Released schema lock
0926 01:55:58.284600 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:58.284863 (+ 263us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:58.284886 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":120811,"reported_tablets":15,"spinlock_wait_cycles":51157760,"child_traces":[["op",{"apply.queue_time_us":15067,"apply.run_cpu_time_us":1488,"apply.run_wall_time_us":1495,"num_ops":1,"prepare.queue_time_us":125,"prepare.run_cpu_time_us":2408,"prepare.run_wall_time_us":7543,"raft.queue_time_us":86,"raft.run_cpu_time_us":675,"raft.run_wall_time_us":674,"replication_time_us":26675}]]}
I20260926 01:56:01.235792 8874 tablet_service.cc:1511] Processing CreateTablet for tablet b85e2acbec2e485cb1b9728bd5763b0e (DEFAULT_TABLE table=default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.236617 8534 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_d8e349dae18c4eb3ae3be35dc46bf47c" } new_table_name: "default.Table_f8b2b30433ac4b2c95082c4b0199bce9" modify_external_catalogs: true
I20260926 01:56:01.237027 8874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85e2acbec2e485cb1b9728bd5763b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.236825 9046 tablet_service.cc:1511] Processing CreateTablet for tablet b85e2acbec2e485cb1b9728bd5763b0e (DEFAULT_TABLE table=default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.237982 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85e2acbec2e485cb1b9728bd5763b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.235846 8879 tablet_service.cc:1511] Processing CreateTablet for tablet c450d101aa2b47c0bac3d4964c7bd6bd (DEFAULT_TABLE table=default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.239660 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c450d101aa2b47c0bac3d4964c7bd6bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:01.143213 8535 rpcz_store.cc:269] 0926 01:55:57.960487 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.961983 (+ 1496us) service_pool.cc:224] Handling call
0926 01:55:58.051717 (+ 89734us) spinlock_profiling.cc:243] Waited 85.6 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:01.142669 (+3090952us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:55:59.684819 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:55:59.685315 (+ 496us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:55:59.685344 (+ 29us) write_op.cc:435] Acquired schema lock
0926 01:55:59.685356 (+ 12us) tablet.cc:662] Decoding operations
0926 01:55:59.685667 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:55:59.685733 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0926 01:55:59.685759 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:55:59.685939 (+ 180us) tablet.cc:701] Row locks acquired
0926 01:55:59.685958 (+ 19us) write_op.cc:260] PREPARE: finished
0926 01:55:59.686093 (+ 135us) write_op.cc:270] Start()
0926 01:55:59.686191 (+ 98us) write_op.cc:276] Timestamp: P: 1790387759686036 usec, L: 0
0926 01:55:59.686211 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:55:59.686901 (+ 690us) log.cc:844] Serialized 530 byte log entry
0926 01:55:59.706102 (+ 19201us) op_driver.cc:464] REPLICATION: finished
0926 01:55:59.709998 (+ 3896us) write_op.cc:301] APPLY: starting
0926 01:55:59.710062 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0926 01:55:59.710163 (+ 101us) tablet.cc:1397] finished BulkCheckPresence
0926 01:55:59.710272 (+ 109us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:55:59.710446 (+ 174us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:55:59.710598 (+ 152us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:55:59.710643 (+ 45us) write_op.cc:312] APPLY: finished
0926 01:55:59.710844 (+ 201us) log.cc:844] Serialized 23 byte log entry
0926 01:55:59.710994 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:59.711063 (+ 69us) write_op.cc:454] Released schema lock
0926 01:55:59.711096 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0926 01:55:59.714745 (+ 3649us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:55:59.714775 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"spinlock_wait_cycles":196980480,"child_traces":[["op",{"apply.queue_time_us":3715,"apply.run_cpu_time_us":1315,"apply.run_wall_time_us":1315,"num_ops":1,"prepare.queue_time_us":107710,"prepare.run_cpu_time_us":2506,"prepare.run_wall_time_us":29894,"raft.queue_time_us":136,"raft.run_cpu_time_us":815,"raft.run_wall_time_us":815,"replication_time_us":19790}]]}
I20260926 01:56:01.243193 9051 tablet_service.cc:1511] Processing CreateTablet for tablet c450d101aa2b47c0bac3d4964c7bd6bd (DEFAULT_TABLE table=default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.244204 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c450d101aa2b47c0bac3d4964c7bd6bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.243976 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:01.245580 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:56:01.138036 8537 rpcz_store.cc:269] 0926 01:55:59.817661 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.820499 (+ 2838us) service_pool.cc:224] Handling call
0926 01:55:59.872610 (+ 52111us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:01.136807 (+1264197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69229,"reported_tablets":23,"spinlock_wait_cycles":62632832}
I20260926 01:56:01.251889 8537 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_2b020c7bf3ca40f5932fadfe0dc566d4" } new_table_name: "default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0" modify_external_catalogs: true
I20260926 01:56:01.196394 9048 tablet_service.cc:1511] Processing CreateTablet for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (DEFAULT_TABLE table=default.Table_86c054848f05455c849277b21ead1bd7 [id=7eccc1870b21429ea89fa66914b317dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.254141 9048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:01.260989 8533 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 288) took 1150 ms. Trace:
W20260926 01:56:01.149008 8532 rpcz_store.cc:269] 0926 01:55:59.943977 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.945094 (+ 1117us) service_pool.cc:224] Handling call
0926 01:55:59.962303 (+ 17209us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:01.148260 (+1185957us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:00.520836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:00.521368 (+ 532us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:00.521398 (+ 30us) write_op.cc:435] Acquired schema lock
0926 01:56:00.521411 (+ 13us) tablet.cc:662] Decoding operations
0926 01:56:00.521738 (+ 327us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:00.521787 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:00.521814 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:00.521995 (+ 181us) tablet.cc:701] Row locks acquired
0926 01:56:00.522012 (+ 17us) write_op.cc:260] PREPARE: finished
0926 01:56:00.522154 (+ 142us) write_op.cc:270] Start()
0926 01:56:00.522256 (+ 102us) write_op.cc:276] Timestamp: P: 1790387760522116 usec, L: 0
0926 01:56:00.522277 (+ 21us) op_driver.cc:348] REPLICATION: starting
0926 01:56:00.522933 (+ 656us) log.cc:844] Serialized 356 byte log entry
0926 01:56:00.531914 (+ 8981us) op_driver.cc:464] REPLICATION: finished
0926 01:56:00.532291 (+ 377us) write_op.cc:301] APPLY: starting
0926 01:56:00.532358 (+ 67us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:00.532460 (+ 102us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:00.532579 (+ 119us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:00.532751 (+ 172us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:00.532885 (+ 134us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:00.532928 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:56:00.533124 (+ 196us) log.cc:844] Serialized 23 byte log entry
0926 01:56:00.533313 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.533415 (+ 102us) write_op.cc:454] Released schema lock
0926 01:56:00.533461 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:00.533813 (+ 352us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.533843 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28735,"reported_tablets":20,"spinlock_wait_cycles":25440000,"child_traces":[["op",{"apply.queue_time_us":198,"apply.run_cpu_time_us":1453,"apply.run_wall_time_us":1452,"num_ops":1,"prepare.queue_time_us":177,"prepare.run_cpu_time_us":2424,"prepare.run_wall_time_us":2424,"raft.queue_time_us":195,"raft.run_cpu_time_us":762,"raft.run_wall_time_us":762,"replication_time_us":9528}]]}
I20260926 01:56:01.268374 9188 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.266417 8532 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_818d89431f354c43ac77ba53fcb9cd8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:56:01.268489 8540 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet cbf8818e65d74a32ba21dcb8a9a56a62
I20260926 01:56:01.284549 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17b8b0d54f5645e5963f1de217f94c84" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:01.284869 8540 catalog_manager.cc:6934] ReplaceTablet: tablet cbf8818e65d74a32ba21dcb8a9a56a62 deleted and replaced by tablet edb7afa012944b3ba3c34bd994a95a83
I20260926 01:56:01.285218 8540 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 289) took 1051 ms. Trace:
I20260926 01:56:01.285801 9053 tablet_service.cc:1558] Processing DeleteTablet for tablet cbf8818e65d74a32ba21dcb8a9a56a62 with delete_type TABLET_DATA_DELETED (replaced by tablet edb7afa012944b3ba3c34bd994a95a83) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:01.286599 10130 raft_consensus.cc:493] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc00feec3f564608851f0112e97ab532)
I20260926 01:56:01.287076 10130 raft_consensus.cc:515] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:01.285487 8540 rpcz_store.cc:276] 0926 01:56:00.233941 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.065229 (+831288us) service_pool.cc:224] Handling call
0926 01:56:01.267762 (+202533us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.267789 (+ 27us) write_op.cc:454] Released schema lock
0926 01:56:01.285153 (+ 17364us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.175954 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.176439 (+ 485us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.176469 (+ 30us) write_op.cc:435] Acquired schema lock
0926 01:56:01.176482 (+ 13us) tablet.cc:662] Decoding operations
0926 01:56:01.176854 (+ 372us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.176903 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.176938 (+ 35us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:01.177177 (+ 239us) tablet.cc:701] Row locks acquired
0926 01:56:01.177196 (+ 19us) write_op.cc:260] PREPARE: finished
0926 01:56:01.177340 (+ 144us) write_op.cc:270] Start()
0926 01:56:01.177442 (+ 102us) write_op.cc:276] Timestamp: P: 1790387761177278 usec, L: 0
0926 01:56:01.177459 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.178043 (+ 584us) log.cc:844] Serialized 659 byte log entry
0926 01:56:01.186807 (+ 8764us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.187311 (+ 504us) write_op.cc:301] APPLY: starting
0926 01:56:01.187380 (+ 69us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:01.187506 (+ 126us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:01.187617 (+ 111us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:01.187994 (+ 377us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:01.188236 (+ 242us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:01.188284 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:56:01.188491 (+ 207us) log.cc:844] Serialized 29 byte log entry
0926 01:56:01.188661 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.188746 (+ 85us) write_op.cc:454] Released schema lock
0926 01:56:01.188781 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":322,"apply.run_cpu_time_us":1668,"apply.run_wall_time_us":1669,"num_ops":2,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":2421,"prepare.run_wall_time_us":2422,"raft.queue_time_us":219,"raft.run_cpu_time_us":722,"raft.run_wall_time_us":723,"replication_time_us":9238}]]}
I20260926 01:56:01.288977 8540 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_fe07ca51a9dd47b5a78a217875523300" } modify_external_catalogs: true
I20260926 01:56:01.289517 8540 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_fe07ca51a9dd47b5a78a217875523300" } modify_external_catalogs: true
W20260926 01:56:01.299016 9006 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:01.299455 9006 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:01.301167 10127 raft_consensus.cc:2749] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:01.312088 8538 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 8d198e51b88948548cfd95dd893a26ef
I20260926 01:56:01.316303 10130 leader_election.cc:290] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:01.318748 10137 raft_consensus.cc:493] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:01.329430 9711 raft_consensus.cc:2243] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:01.330013 9711 raft_consensus.cc:2272] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:01.332180 9711 ts_tablet_manager.cc:1916] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:01.333105 9006 leader_election.cc:343] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:56:01.340859 8531 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:01.342869 8531 catalog_manager.cc:6934] ReplaceTablet: tablet 1f6d9fedc7b241b59b25d591c7dc2fea deleted and replaced by tablet af0840d6761a4521b0b6136b80eb6a6c
I20260926 01:56:01.343379 8531 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 290) took 1051 ms. Trace:
I20260926 01:56:01.356526 9049 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d198e51b88948548cfd95dd893a26ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:01.347165 8531 rpcz_store.cc:276] 0926 01:56:00.292077 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.168191 (+876114us) service_pool.cc:224] Handling call
0926 01:56:01.343316 (+175125us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.309658 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.310190 (+ 532us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.310232 (+ 42us) write_op.cc:435] Acquired schema lock
0926 01:56:01.310246 (+ 14us) tablet.cc:662] Decoding operations
0926 01:56:01.310659 (+ 413us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.310711 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.310743 (+ 32us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:01.310991 (+ 248us) tablet.cc:701] Row locks acquired
0926 01:56:01.311009 (+ 18us) write_op.cc:260] PREPARE: finished
0926 01:56:01.318647 (+ 7638us) write_op.cc:270] Start()
0926 01:56:01.318763 (+ 116us) write_op.cc:276] Timestamp: P: 1790387761318558 usec, L: 0
0926 01:56:01.318786 (+ 23us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.319552 (+ 766us) log.cc:844] Serialized 659 byte log entry
0926 01:56:01.333394 (+ 13842us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.333769 (+ 375us) write_op.cc:301] APPLY: starting
0926 01:56:01.333828 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:01.333949 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:01.334033 (+ 84us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:01.334685 (+ 652us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:01.334931 (+ 246us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:01.334977 (+ 46us) write_op.cc:312] APPLY: finished
0926 01:56:01.335199 (+ 222us) log.cc:844] Serialized 29 byte log entry
0926 01:56:01.335372 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.335452 (+ 80us) write_op.cc:454] Released schema lock
0926 01:56:01.335485 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":1756,"apply.run_wall_time_us":1921,"mutex_wait_us":58502,"num_ops":2,"prepare.queue_time_us":58639,"prepare.run_cpu_time_us":2782,"prepare.run_wall_time_us":50616,"raft.queue_time_us":1791,"raft.run_cpu_time_us":856,"raft.run_wall_time_us":857,"replication_time_us":14505}]]}
W20260926 01:56:01.382682 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 147218us
I20260926 01:56:01.385079 8877 tablet_service.cc:1558] Processing DeleteTablet for tablet cbf8818e65d74a32ba21dcb8a9a56a62 with delete_type TABLET_DATA_DELETED (replaced by tablet edb7afa012944b3ba3c34bd994a95a83) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.390887 8877 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d198e51b88948548cfd95dd893a26ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.392886 9339 tablet_bootstrap.cc:492] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:01.393404 9339 ts_tablet_manager.cc:1403] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.495s user 0.005s sys 0.010s
I20260926 01:56:01.397646 9045 tablet_service.cc:1558] Processing DeleteTablet for tablet 1f6d9fedc7b241b59b25d591c7dc2fea with delete_type TABLET_DATA_DELETED (replaced by tablet af0840d6761a4521b0b6136b80eb6a6c) from {username='slave'} at 127.0.0.1:51458
W20260926 01:56:01.406406 9003 leader_election.cc:343] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:56:01.406921 9003 leader_election.cc:304] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:01.407665 10143 raft_consensus.cc:2749] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:01.420425 9339 raft_consensus.cc:359] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:56:01.346081 8539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 35) took 1230 ms (client timeout 1000 ms). Trace:
I20260926 01:56:01.426712 10090 raft_consensus.cc:493] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:01.427189 10090 raft_consensus.cc:515] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:01.429129 10090 leader_election.cc:290] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:01.426589 8539 rpcz_store.cc:269] 0926 01:56:00.115734 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.263070 (+147336us) service_pool.cc:224] Handling call
0926 01:56:00.974617 (+711547us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.974647 (+ 30us) write_op.cc:454] Released schema lock
0926 01:56:01.345997 (+371350us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:00.841698 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:00.842210 (+ 512us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:00.842242 (+ 32us) write_op.cc:435] Acquired schema lock
0926 01:56:00.842259 (+ 17us) tablet.cc:662] Decoding operations
0926 01:56:00.842844 (+ 585us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:00.842904 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:00.842934 (+ 30us) tablet.cc:685] Acquiring locks for 3 operations
0926 01:56:00.843268 (+ 334us) tablet.cc:701] Row locks acquired
0926 01:56:00.843287 (+ 19us) write_op.cc:260] PREPARE: finished
0926 01:56:00.843449 (+ 162us) write_op.cc:270] Start()
0926 01:56:00.843572 (+ 123us) write_op.cc:276] Timestamp: P: 1790387760843376 usec, L: 0
0926 01:56:00.843594 (+ 22us) op_driver.cc:348] REPLICATION: starting
0926 01:56:00.844324 (+ 730us) log.cc:844] Serialized 1284 byte log entry
0926 01:56:00.856017 (+ 11693us) op_driver.cc:464] REPLICATION: finished
0926 01:56:00.856392 (+ 375us) write_op.cc:301] APPLY: starting
0926 01:56:00.856453 (+ 61us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:00.856598 (+ 145us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:00.856699 (+ 101us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:00.857060 (+ 361us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:00.857219 (+ 159us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0926 01:56:00.857266 (+ 47us) write_op.cc:312] APPLY: finished
0926 01:56:00.857481 (+ 215us) log.cc:844] Serialized 35 byte log entry
0926 01:56:00.857632 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.857711 (+ 79us) write_op.cc:454] Released schema lock
0926 01:56:00.857749 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":30225,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":227,"apply.run_cpu_time_us":1537,"apply.run_wall_time_us":1536,"num_ops":3,"prepare.queue_time_us":6808,"prepare.run_cpu_time_us":2971,"prepare.run_wall_time_us":5247,"raft.queue_time_us":1390,"raft.run_cpu_time_us":810,"raft.run_wall_time_us":810,"replication_time_us":12313}]]}
I20260926 01:56:01.431604 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b0e40e5e964b90bd2dd158bbc4497a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:01.441696 9070 raft_consensus.cc:2468] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:01.442619 9339 raft_consensus.cc:385] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:01.442935 9339 raft_consensus.cc:740] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:01.443216 8843 leader_election.cc:304] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:01.443660 9339 consensus_queue.cc:260] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:01.444367 10090 raft_consensus.cc:2804] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:01.444732 10090 raft_consensus.cc:493] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:01.445053 10090 raft_consensus.cc:3060] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:01.447009 9339 ts_tablet_manager.cc:1434] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.053s user 0.007s sys 0.001s
I20260926 01:56:01.261341 8533 rpcz_store.cc:276] 0926 01:56:00.110363 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.048987 (+938624us) service_pool.cc:224] Handling call
0926 01:56:01.061491 (+ 12504us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:01.206915 (+145424us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:01.259513 (+ 52598us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.259543 (+ 30us) write_op.cc:454] Released schema lock
0926 01:56:01.260317 (+ 774us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:01.260630 (+ 313us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:01.260920 (+ 290us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.209128 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.209644 (+ 516us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.209675 (+ 31us) write_op.cc:435] Acquired schema lock
0926 01:56:01.209693 (+ 18us) tablet.cc:662] Decoding operations
0926 01:56:01.210037 (+ 344us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.210090 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.210118 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:01.210372 (+ 254us) tablet.cc:701] Row locks acquired
0926 01:56:01.210393 (+ 21us) write_op.cc:260] PREPARE: finished
0926 01:56:01.210547 (+ 154us) write_op.cc:270] Start()
0926 01:56:01.210651 (+ 104us) write_op.cc:276] Timestamp: P: 1790387761210485 usec, L: 0
0926 01:56:01.210671 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.211404 (+ 733us) log.cc:844] Serialized 433 byte log entry
0926 01:56:01.220518 (+ 9114us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.220881 (+ 363us) write_op.cc:301] APPLY: starting
0926 01:56:01.220947 (+ 66us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:01.221067 (+ 120us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:01.221170 (+ 103us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:01.221589 (+ 419us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:01.221733 (+ 144us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:01.221781 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:56:01.222001 (+ 220us) log.cc:844] Serialized 29 byte log entry
0926 01:56:01.222196 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.222281 (+ 85us) write_op.cc:454] Released schema lock
0926 01:56:01.222317 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":215,"apply.run_cpu_time_us":1652,"apply.run_wall_time_us":1651,"num_ops":2,"prepare.queue_time_us":118,"prepare.run_cpu_time_us":2547,"prepare.run_wall_time_us":3793,"raft.queue_time_us":217,"raft.run_cpu_time_us":785,"raft.run_wall_time_us":784,"replication_time_us":9727}]]}
I20260926 01:56:01.450961 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 1f6d9fedc7b241b59b25d591c7dc2fea with delete_type TABLET_DATA_DELETED (replaced by tablet af0840d6761a4521b0b6136b80eb6a6c) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:01.458422 10090 raft_consensus.cc:515] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:01.460423 10048 consensus_queue.cc:1048] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:01.465230 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b0e40e5e964b90bd2dd158bbc4497a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:01.465797 9072 raft_consensus.cc:3060] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:01.474169 9072 raft_consensus.cc:2468] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:01.475867 8843 leader_election.cc:304] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:01.476962 10048 raft_consensus.cc:2804] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:01.477895 10048 raft_consensus.cc:697] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:01.479193 10048 consensus_queue.cc:237] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:01.493691 10090 leader_election.cc:290] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:01.282940 8532 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_818d89431f354c43ac77ba53fcb9cd8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:01.505203 9711 ts_tablet_manager.cc:1929] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:01.505752 9711 log.cc:1199] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/55452e70a7fe48efbafdc7cbb29c6272
I20260926 01:56:01.512583 9339 tablet_bootstrap.cc:492] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:01.537945 9340 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.023s sys 0.025s
W20260926 01:56:01.538441 9340 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.508s user 0.024s sys 0.025s
I20260926 01:56:01.539024 9711 ts_tablet_manager.cc:1950] T 55452e70a7fe48efbafdc7cbb29c6272 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:01.541543 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 164) took 7875 ms. Trace:
I20260926 01:56:01.541935 9711 rpcz_store.cc:276] 0926 01:55:53.666405 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:53.666564 (+ 159us) service_pool.cc:224] Handling call
0926 01:56:00.883803 (+7217239us) ts_tablet_manager.cc:1151] Deleting tablet 55452e70a7fe48efbafdc7cbb29c6272
0926 01:56:00.883868 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.373930 (+490062us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.505048 (+131118us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.541477 (+ 36429us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":397949,"tablet-delete.queue_time_us":7216633}
I20260926 01:56:01.560925 10127 raft_consensus.cc:493] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:01.567770 10127 raft_consensus.cc:515] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:01.568621 9340 tablet_bootstrap.cc:492] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:01.569113 9340 ts_tablet_manager.cc:1403] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.548s user 0.030s sys 0.025s
I20260926 01:56:01.570003 10127 leader_election.cc:290] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:01.572352 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 55452e70a7fe48efbafdc7cbb29c6272 (table default.Table_cdcfa57f52a747aaa93bb04410df7061 [id=fccbbbc294bb4548813a1229bd9e2ef8]) successfully deleted
I20260926 01:56:01.573427 9711 tablet_replica.cc:333] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:01.574394 9711 raft_consensus.cc:2243] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:01.575090 9711 raft_consensus.cc:2272] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:01.577430 9711 ts_tablet_manager.cc:1916] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:01.577905 10123 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 28713133360477908) tablet 00000000000000000000000000000000 cur holder 0x7b5800040500
I20260926 01:56:01.608500 8537 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_79241696f6954afba998044c7a981755" } modify_external_catalogs: true
I20260926 01:56:01.609354 8537 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_79241696f6954afba998044c7a981755" } modify_external_catalogs: true
W20260926 01:56:01.610821 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 349511us
I20260926 01:56:01.611914 10186 tablet_replica.cc:333] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:01.611975 9340 raft_consensus.cc:359] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:01.612726 9340 raft_consensus.cc:385] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:01.613018 9340 raft_consensus.cc:740] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:01.614243 9340 consensus_queue.cc:260] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:01.615684 10190 tablet.cc:1751] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:56:01.319288 10137 raft_consensus.cc:515] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:01.619144 10137 leader_election.cc:290] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:01.622113 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet ff004cf835c34095a836ad683cec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:58 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.624389 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.625356 9196 tablet_service.cc:1558] Processing DeleteTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:52 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.582257 9339 tablet_bootstrap.cc:654] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:01.627110 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (DEFAULT_TABLE table=default.Table_86c054848f05455c849277b21ead1bd7 [id=7eccc1870b21429ea89fa66914b317dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.628238 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:01.636972 9187 consensus_meta.cc:287] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.565s user 0.010s sys 0.015s
I20260926 01:56:01.643710 10186 raft_consensus.cc:2243] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:01.644753 10186 raft_consensus.cc:2272] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:01.647377 9341 tablet_bootstrap.cc:654] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:01.649446 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:01.650123 9072 raft_consensus.cc:2468] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:56:01.656632 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_c5ca9108f53e46bbb3d6acf830b83ce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:01.658484 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:01.665584 10199 tablet.cc:1751] T f1f020590a8b428788e2bb516a11cc45 P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:56:01.673925 9339 tablet_bootstrap.cc:492] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:01.674412 9339 ts_tablet_manager.cc:1403] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.002s
I20260926 01:56:01.677050 9339 raft_consensus.cc:359] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:01.677690 9339 raft_consensus.cc:385] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:01.677942 9339 raft_consensus.cc:740] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:01.647881 10186 ts_tablet_manager.cc:1916] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:01.648845 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 269058us
I20260926 01:56:01.650789 9192 tablet_service.cc:1511] Processing CreateTablet for tablet c450d101aa2b47c0bac3d4964c7bd6bd (DEFAULT_TABLE table=default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.653123 9188 tablet_service.cc:1558] Processing DeleteTablet for tablet cbf8818e65d74a32ba21dcb8a9a56a62 with delete_type TABLET_DATA_DELETED (replaced by tablet edb7afa012944b3ba3c34bd994a95a83) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.654019 9198 tablet_service.cc:1558] Processing DeleteTablet for tablet 1f6d9fedc7b241b59b25d591c7dc2fea with delete_type TABLET_DATA_DELETED (replaced by tablet af0840d6761a4521b0b6136b80eb6a6c) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.654804 9197 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d198e51b88948548cfd95dd893a26ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:01.671473 9155 leader_election.cc:304] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:56:01.673312 9195 consensus_meta.cc:287] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.604s user 0.007s sys 0.000s
I20260926 01:56:01.701414 10192 raft_consensus.cc:2804] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:01.702271 10192 raft_consensus.cc:493] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:01.706743 10192 raft_consensus.cc:3060] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:01.712743 9187 tablet_service.cc:1511] Processing CreateTablet for tablet b85e2acbec2e485cb1b9728bd5763b0e (DEFAULT_TABLE table=default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:01.737033 10127 consensus_queue.cc:1048] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:01.739094 9339 consensus_queue.cc:260] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:01.739846 9003 leader_election.cc:343] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: b659eca3c97144aaacd41e747aecd96d
I20260926 01:56:01.744307 10130 consensus_queue.cc:1048] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:01.745612 9339 ts_tablet_manager.cc:1434] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.071s user 0.004s sys 0.001s
W20260926 01:56:01.747908 9006 leader_election.cc:343] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: b659eca3c97144aaacd41e747aecd96d
I20260926 01:56:01.748363 9006 leader_election.cc:304] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:01.749022 10143 raft_consensus.cc:2749] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:01.750201 9340 ts_tablet_manager.cc:1434] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.181s user 0.007s sys 0.000s
I20260926 01:56:01.752688 9711 ts_tablet_manager.cc:1929] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:01.753087 9711 log.cc:1199] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/a601d850375c43679dde7a46cfe71d22
I20260926 01:56:01.761791 9339 tablet_bootstrap.cc:492] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:01.770720 9711 ts_tablet_manager.cc:1950] T a601d850375c43679dde7a46cfe71d22 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:01.772224 8539 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_bf5ded4035ed4595a340d90cab2ba64c" } new_table_name: "default.Table_a21182bd8e5043a99c406b2f1ed0f766" modify_external_catalogs: true
I20260926 01:56:01.773034 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 167) took 7755 ms. Trace:
I20260926 01:56:01.773393 9711 rpcz_store.cc:276] 0926 01:55:54.017096 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:54.018522 (+ 1426us) service_pool.cc:224] Handling call
0926 01:56:01.573223 (+7554701us) ts_tablet_manager.cc:1151] Deleting tablet a601d850375c43679dde7a46cfe71d22
0926 01:56:01.573289 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.733265 (+159976us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.752539 (+ 19274us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.772962 (+ 20423us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7554139}
I20260926 01:56:01.779785 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 174) took 6635 ms. Trace:
I20260926 01:56:01.780233 9711 rpcz_store.cc:276] 0926 01:55:55.144343 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.149400 (+ 5057us) service_pool.cc:224] Handling call
0926 01:56:01.779345 (+6629945us) ts_tablet_manager.cc:1151] Deleting tablet 0c6cccff3de64f3384ce58bb84c43487
0926 01:56:01.779413 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.779728 (+ 315us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6629314}
I20260926 01:56:01.783030 9711 tablet_replica.cc:333] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:01.778669 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:01.811013 9340 tablet_bootstrap.cc:492] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:01.779094 10199 tablet.cc:1751] T 28eaf7381f364a2682dd07f2bd90ede3 P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:01.793874 10127 consensus_queue.cc:1048] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:01.798672 9339 tablet_bootstrap.cc:654] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:01.829298 9711 raft_consensus.cc:2243] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:01.738739 10134 consensus_queue.cc:1048] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:01.774649 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet a601d850375c43679dde7a46cfe71d22 (table default.Table_caaf07e0f7d24f7388b358362fd1668f [id=6b7b2329cccd4e96acfc168a00029171]) successfully deleted
W20260926 01:56:01.835299 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 60669.8us
W20260926 01:56:01.836411 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 0c6cccff3de64f3384ce58bb84c43487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c6cccff3de64f3384ce58bb84c43487 already in progress: creating tablet
I20260926 01:56:01.817822 8536 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 287) took 1743 ms. Trace:
I20260926 01:56:01.836962 8536 rpcz_store.cc:276] 0926 01:56:00.074534 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.879928 (+805394us) service_pool.cc:224] Handling call
0926 01:56:01.042509 (+162581us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:01.269084 (+226575us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:01.721739 (+452655us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:01.817414 (+ 95675us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:01.817749 (+ 335us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.605412 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.606855 (+ 1443us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.606887 (+ 32us) write_op.cc:435] Acquired schema lock
0926 01:56:01.606900 (+ 13us) tablet.cc:662] Decoding operations
0926 01:56:01.701798 (+ 94898us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.701857 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.701888 (+ 31us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:01.702173 (+ 285us) tablet.cc:701] Row locks acquired
0926 01:56:01.702189 (+ 16us) write_op.cc:260] PREPARE: finished
0926 01:56:01.702353 (+ 164us) write_op.cc:270] Start()
0926 01:56:01.702493 (+ 140us) write_op.cc:276] Timestamp: P: 1790387761702290 usec, L: 0
0926 01:56:01.702524 (+ 31us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.703155 (+ 631us) log.cc:844] Serialized 433 byte log entry
0926 01:56:01.717621 (+ 14466us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.717994 (+ 373us) write_op.cc:301] APPLY: starting
0926 01:56:01.718063 (+ 69us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:01.718184 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:01.718319 (+ 135us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:01.718904 (+ 585us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:01.719074 (+ 170us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:01.719122 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:56:01.719338 (+ 216us) log.cc:844] Serialized 29 byte log entry
0926 01:56:01.719500 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.719594 (+ 94us) write_op.cc:454] Released schema lock
0926 01:56:01.719635 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:01.771501 (+ 51866us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.771547 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":218,"apply.run_cpu_time_us":1858,"apply.run_wall_time_us":1857,"mutex_wait_us":43380,"num_ops":2,"prepare.queue_time_us":334531,"prepare.run_cpu_time_us":3823,"prepare.run_wall_time_us":166049,"raft.queue_time_us":111,"raft.run_cpu_time_us":816,"raft.run_wall_time_us":816,"replication_time_us":14976}]]}
I20260926 01:56:01.814898 9812 ts_tablet_manager.cc:1929] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:01.846041 9812 log.cc:1199] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/ff004cf835c34095a836ad683cec9256
I20260926 01:56:01.846834 9812 ts_tablet_manager.cc:1950] T ff004cf835c34095a836ad683cec9256 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:01.847697 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 237) took 2071 ms. Trace:
I20260926 01:56:01.848001 9812 rpcz_store.cc:276] 0926 01:55:59.776483 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.777098 (+ 615us) service_pool.cc:224] Handling call
0926 01:56:00.491769 (+714671us) ts_tablet_manager.cc:1151] Deleting tablet ff004cf835c34095a836ad683cec9256
0926 01:56:00.602735 (+110966us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.277251 (+674516us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.814749 (+537498us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.847629 (+ 32880us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":714066}
I20260926 01:56:01.849478 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet ff004cf835c34095a836ad683cec9256 (table default.Table_0818779c2b1449eaaee8c359e22c3b86 [id=5db31752ef1c4ff992bed8d186ba574e]) successfully deleted
I20260926 01:56:01.849406 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 238) took 1588 ms. Trace:
I20260926 01:56:01.850136 9812 rpcz_store.cc:276] 0926 01:56:00.261117 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.261281 (+ 164us) service_pool.cc:224] Handling call
0926 01:56:01.849122 (+1587841us) ts_tablet_manager.cc:1151] Deleting tablet 2156ef6b24c04ea38181832e7f18ff49
0926 01:56:01.849165 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.849354 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587238}
I20260926 01:56:01.852180 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 239) took 1534 ms. Trace:
I20260926 01:56:01.852451 9812 rpcz_store.cc:276] 0926 01:56:00.317496 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.317643 (+ 147us) service_pool.cc:224] Handling call
0926 01:56:01.851748 (+1534105us) ts_tablet_manager.cc:1151] Deleting tablet 2156ef6b24c04ea38181832e7f18ff49
0926 01:56:01.851801 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.852082 (+ 281us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533529}
I20260926 01:56:01.853515 9812 tablet_replica.cc:333] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:01.854259 9812 raft_consensus.cc:2243] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 01:56:01.854266 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 2156ef6b24c04ea38181832e7f18ff49 because the tablet was not found. No further retry: Not found: Tablet not found: 2156ef6b24c04ea38181832e7f18ff49
I20260926 01:56:01.854790 9812 raft_consensus.cc:2272] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:56:01.855003 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 2156ef6b24c04ea38181832e7f18ff49 because the tablet was not found. No further retry: Not found: Tablet not found: 2156ef6b24c04ea38181832e7f18ff49
I20260926 01:56:01.856765 9812 ts_tablet_manager.cc:1916] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:01.865293 8534 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70d2447335b34731a4cb2cd0f3b4bf13 has no consensus state
I20260926 01:56:01.865901 8534 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 292) took 1270 ms. Trace:
I20260926 01:56:01.867113 9711 raft_consensus.cc:2272] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:01.867821 10143 raft_consensus.cc:493] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:01.868234 10143 raft_consensus.cc:515] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:01.869539 8882 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58226 (request call id 244) took 1069 ms. Trace:
I20260926 01:56:01.869804 9711 ts_tablet_manager.cc:1916] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:01.869884 8882 rpcz_store.cc:276] 0926 01:56:00.800575 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.802061 (+ 1486us) service_pool.cc:224] Handling call
0926 01:56:00.940549 (+138488us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:56:00.940649 (+ 100us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:56:01.440290 (+499641us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.869051 (+428761us) tablet_replica.cc:206] Creating consensus instance
0926 01:56:01.869470 (+ 419us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:01.866212 8534 rpcz_store.cc:276] 0926 01:56:00.595734 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.236251 (+640517us) service_pool.cc:224] Handling call
0926 01:56:01.237235 (+ 984us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:56:01.237910 (+ 675us) catalog_manager.cc:3764] Apply alter schema
0926 01:56:01.278565 (+ 40655us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:56:01.291110 (+ 12545us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:56:01.864699 (+573589us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:56:01.865832 (+ 1133us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.771866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.772380 (+ 514us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.772407 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:56:01.772424 (+ 17us) tablet.cc:662] Decoding operations
0926 01:56:01.772750 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.772798 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.772825 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:01.773014 (+ 189us) tablet.cc:701] Row locks acquired
0926 01:56:01.773028 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:56:01.773174 (+ 146us) write_op.cc:270] Start()
0926 01:56:01.773305 (+ 131us) write_op.cc:276] Timestamp: P: 1790387761773106 usec, L: 0
0926 01:56:01.773320 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.773969 (+ 649us) log.cc:844] Serialized 405 byte log entry
0926 01:56:01.862912 (+ 88943us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.863188 (+ 276us) write_op.cc:301] APPLY: starting
0926 01:56:01.863255 (+ 67us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:01.863373 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:01.863479 (+ 106us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:01.863652 (+ 173us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:01.863798 (+ 146us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:01.863841 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:56:01.864049 (+ 208us) log.cc:844] Serialized 23 byte log entry
0926 01:56:01.864205 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.864275 (+ 70us) write_op.cc:454] Released schema lock
0926 01:56:01.864314 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"apply.run_cpu_time_us":1333,"apply.run_wall_time_us":1332,"mutex_wait_us":54163,"num_ops":1,"prepare.queue_time_us":479397,"prepare.run_cpu_time_us":2578,"prepare.run_wall_time_us":73272,"raft.queue_time_us":139,"raft.run_cpu_time_us":775,"raft.run_wall_time_us":775,"replication_time_us":89468,"spinlock_wait_cycles":27008}]]}
I20260926 01:56:01.892743 9340 tablet_bootstrap.cc:654] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:01.898803 10143 leader_election.cc:290] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:01.900981 9003 leader_election.cc:343] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: ff004cf835c34095a836ad683cec9256
I20260926 01:56:01.912040 8535 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 293) took 1224 ms. Trace:
I20260926 01:56:01.912540 8535 rpcz_store.cc:276] 0926 01:56:00.688074 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.243557 (+555483us) service_pool.cc:224] Handling call
0926 01:56:01.245989 (+ 2432us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:01.596149 (+350160us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:01.905693 (+309544us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:01.911649 (+ 5956us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:01.911965 (+ 316us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.845319 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.845805 (+ 486us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.845832 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:56:01.845850 (+ 18us) tablet.cc:662] Decoding operations
0926 01:56:01.846220 (+ 370us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.846269 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.846302 (+ 33us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:01.846563 (+ 261us) tablet.cc:701] Row locks acquired
0926 01:56:01.846583 (+ 20us) write_op.cc:260] PREPARE: finished
0926 01:56:01.846729 (+ 146us) write_op.cc:270] Start()
0926 01:56:01.846835 (+ 106us) write_op.cc:276] Timestamp: P: 1790387761846651 usec, L: 0
0926 01:56:01.846852 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.847484 (+ 632us) log.cc:844] Serialized 433 byte log entry
0926 01:56:01.902632 (+ 55148us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.903213 (+ 581us) write_op.cc:301] APPLY: starting
0926 01:56:01.903273 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:01.903385 (+ 112us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:01.903500 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:01.903960 (+ 460us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:01.904103 (+ 143us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:01.904147 (+ 44us) write_op.cc:312] APPLY: finished
0926 01:56:01.904349 (+ 202us) log.cc:844] Serialized 29 byte log entry
0926 01:56:01.904508 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.904581 (+ 73us) write_op.cc:454] Released schema lock
0926 01:56:01.904624 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:01.904846 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:01.904872 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":427,"apply.run_cpu_time_us":1601,"apply.run_wall_time_us":1601,"num_ops":2,"prepare.queue_time_us":247570,"prepare.run_cpu_time_us":2443,"prepare.run_wall_time_us":2443,"replication_time_us":55657}]]}
I20260926 01:56:01.936358 10199 tablet.cc:1751] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:01.942734 9118 heartbeater.cc:344] Connected to a master server at 127.8.76.188:38143
I20260926 01:56:01.951100 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b0e40e5e964b90bd2dd158bbc4497a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:01.976570 10192 raft_consensus.cc:515] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:56:01.995774 8527 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 880d77d42c394e6a9cf0b3768a915402 (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 880d77d42c394e6a9cf0b3768a915402 consensus state has no leader
I20260926 01:56:02.001597 9339 tablet_bootstrap.cc:492] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.002079 9339 ts_tablet_manager.cc:1403] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.240s user 0.004s sys 0.012s
W20260926 01:56:02.005050 8843 leader_election.cc:343] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:01.837709 10211 tablet.cc:1751] T f1f020590a8b428788e2bb516a11cc45 P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:56:02.027727 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c450d101aa2b47c0bac3d4964c7bd6bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:02.032522 9195 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42408 (request call id 261) took 1201 ms. Trace:
I20260926 01:56:02.033120 9195 rpcz_store.cc:276] 0926 01:56:00.831593 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.831744 (+ 151us) service_pool.cc:224] Handling call
0926 01:56:00.833133 (+ 1389us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:56:00.833218 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:56:01.069311 (+236093us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.701269 (+631958us) tablet_replica.cc:206] Creating consensus instance
0926 01:56:02.032459 (+331190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:02.037623 9339 raft_consensus.cc:359] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.038318 9339 raft_consensus.cc:385] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.040207 9339 raft_consensus.cc:740] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.040906 9339 consensus_queue.cc:260] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:02.044435 10203 log.cc:927] Time spent T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
W20260926 01:56:02.047242 9184 consensus_meta.cc:287] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.962s user 0.004s sys 0.012s
I20260926 01:56:02.049602 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b0e40e5e964b90bd2dd158bbc4497a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:02.049095 8534 master_service.cc:960] ReplaceTablet: received request to replace tablet ff004cf835c34095a836ad683cec9256 from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:02.051808 9190 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42408 (request call id 259) took 1317 ms. Trace:
I20260926 01:56:02.052167 9190 rpcz_store.cc:276] 0926 01:56:00.734233 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.734575 (+ 342us) service_pool.cc:224] Handling call
0926 01:56:00.736122 (+ 1547us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:56:00.736203 (+ 81us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:56:01.260020 (+523817us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.051340 (+791320us) tablet_replica.cc:206] Creating consensus instance
0926 01:56:02.051749 (+ 409us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:02.057695 9339 ts_tablet_manager.cc:1434] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.055s user 0.002s sys 0.006s
I20260926 01:56:02.059041 9339 tablet_bootstrap.cc:492] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:02.062999 10174 consensus_queue.cc:1048] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:02.064322 10217 tablet.cc:1751] T f2ee16b0c3e84936abba8248f194786a P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260926 01:56:02.067340 8843 leader_election.cc:343] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.076965 8534 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_dff0511306454aa1a93fe774b3769823" } new_table_name: "default.Table_8004b577262548dbbc92c297b99e01b2" modify_external_catalogs: true
I20260926 01:56:02.085791 10206 tablet.cc:1751] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:02.088356 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_da5237bb06194562a023680e328f087d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:02.090052 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da5237bb06194562a023680e328f087d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:02.098767 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85e2acbec2e485cb1b9728bd5763b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:02.102636 10190 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42408 (request call id 264) took 1103 ms. Trace:
I20260926 01:56:02.103194 10190 rpcz_store.cc:276] 0926 01:56:00.999725 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.999882 (+ 157us) service_pool.cc:224] Handling call
0926 01:56:02.102521 (+1102639us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.001033 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:56:01.001389 (+ 356us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:56:01.001419 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0926 01:56:01.001466 (+ 47us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:56:01.001590 (+ 124us) alter_schema_op.cc:128] START. Timestamp: 1790387761001495
0926 01:56:01.001605 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.002198 (+ 593us) log.cc:844] Serialized 173 byte log entry
0926 01:56:01.287692 (+285494us) op_driver.cc:464] REPLICATION: finished
0926 01:56:01.615580 (+327888us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:56:01.617159 (+ 1579us) log.cc:844] Serialized 5 byte log entry
0926 01:56:02.024958 (+407799us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.025676 (+ 718us) log.cc:844] Serialized 14 byte log entry
0926 01:56:02.035749 (+ 10073us) alter_schema_op.cc:73] Released schema lock
0926 01:56:02.037622 (+ 1873us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":327672,"prepare.queue_time_us":175,"prepare.run_cpu_time_us":1481,"prepare.run_wall_time_us":1482,"raft.queue_time_us":1414,"raft.run_cpu_time_us":713,"raft.run_wall_time_us":713,"replication_time_us":285952,"thread_start_us":359923,"threads_started":1}]]}
I20260926 01:56:02.112993 10130 raft_consensus.cc:493] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.113626 10130 raft_consensus.cc:515] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.120499 9339 tablet_bootstrap.cc:654] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.120378 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:02.120944 9072 raft_consensus.cc:3060] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:02.121701 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_730735d338704c849d0e9a76c846e6cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:02.123402 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_730735d338704c849d0e9a76c846e6cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:02.118760 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c448caeeedfb4fc88eee88e40eebfeeb" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:02.125775 9003 leader_election.cc:343] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.127770 9072 raft_consensus.cc:2468] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:02.129081 9155 leader_election.cc:304] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:02.152415 9812 ts_tablet_manager.cc:1929] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260926 01:56:02.155488 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 281748us
W20260926 01:56:02.156243 9152 leader_election.cc:343] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.118366 10130 leader_election.cc:290] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:02.179338 10130 raft_consensus.cc:493] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.179962 10130 raft_consensus.cc:515] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.191496 9341 tablet_bootstrap.cc:492] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:02.191648 9812 log.cc:1199] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/3571967749844fd7859cf179a93cf403
I20260926 01:56:02.191809 9339 tablet_bootstrap.cc:492] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.192270 9339 ts_tablet_manager.cc:1403] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.004s
I20260926 01:56:02.192471 9812 ts_tablet_manager.cc:1950] T 3571967749844fd7859cf179a93cf403 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:02.193013 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4623456312284a698ae69d4054909647" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:02.195041 9339 raft_consensus.cc:359] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.195685 9339 raft_consensus.cc:385] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.195938 9339 raft_consensus.cc:740] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.200333 10130 leader_election.cc:290] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:02.206794 9339 consensus_queue.cc:260] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.214713 9339 ts_tablet_manager.cc:1434] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260926 01:56:02.215756 9339 tablet_bootstrap.cc:492] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:02.216759 9711 ts_tablet_manager.cc:1929] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:02.221392 9339 tablet_bootstrap.cc:654] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.233580 9711 log.cc:1199] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/c27f1dd7bb0449a6a3fa371a8636e09b
I20260926 01:56:02.237099 9711 ts_tablet_manager.cc:1950] T c27f1dd7bb0449a6a3fa371a8636e09b P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:02.253696 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 175) took 7098 ms. Trace:
I20260926 01:56:02.254571 9711 rpcz_store.cc:276] 0926 01:55:55.150876 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.162788 (+ 11912us) service_pool.cc:224] Handling call
0926 01:56:01.782855 (+6620067us) ts_tablet_manager.cc:1151] Deleting tablet c27f1dd7bb0449a6a3fa371a8636e09b
0926 01:56:01.782910 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.985805 (+202895us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.216612 (+230807us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.248727 (+ 32115us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6619469}
I20260926 01:56:02.227866 10174 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.256536 10174 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.258930 10174 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:02.229717 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 249) took 1126 ms. Trace:
I20260926 01:56:02.260576 9812 rpcz_store.cc:276] 0926 01:56:01.102872 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.102969 (+ 97us) service_pool.cc:224] Handling call
0926 01:56:01.853363 (+750394us) ts_tablet_manager.cc:1151] Deleting tablet 3571967749844fd7859cf179a93cf403
0926 01:56:01.853402 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:01.950514 (+ 97112us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.152262 (+201748us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.229648 (+ 77386us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":749733}
I20260926 01:56:02.261866 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 250) took 1098 ms. Trace:
I20260926 01:56:02.256877 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet c27f1dd7bb0449a6a3fa371a8636e09b (table default.Table_29ce250e9d9a4162966731d01a8e3b23 [id=e664d2aa0cf74dfbad934ad658e8ec0e]) successfully deleted
I20260926 01:56:02.262171 9812 rpcz_store.cc:276] 0926 01:56:01.163043 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.163171 (+ 128us) service_pool.cc:224] Handling call
0926 01:56:02.261582 (+1098411us) ts_tablet_manager.cc:1151] Deleting tablet 3bf2fa5fbe4146b5adb27ff09de878d3
0926 01:56:02.261641 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.261808 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097853}
I20260926 01:56:02.264294 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 251) took 1101 ms. Trace:
I20260926 01:56:02.264585 9812 rpcz_store.cc:276] 0926 01:56:01.163362 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.163466 (+ 104us) service_pool.cc:224] Handling call
0926 01:56:02.263406 (+1099940us) ts_tablet_manager.cc:1151] Deleting tablet c36ea09cf15647c0a344c9ba9b289d61
0926 01:56:02.263447 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.264243 (+ 796us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099340}
I20260926 01:56:02.266355 9812 tablet_replica.cc:333] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:02.267184 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]) successfully deleted
W20260926 01:56:02.267920 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 because the tablet was not found. No further retry: Not found: Tablet not found: 3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:56:02.267879 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17b8b0d54f5645e5963f1de217f94c84" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:02.268496 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet c36ea09cf15647c0a344c9ba9b289d61 because the tablet was not found. No further retry: Not found: Tablet not found: c36ea09cf15647c0a344c9ba9b289d61
I20260926 01:56:02.268460 9070 raft_consensus.cc:2468] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:02.269524 8843 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:02.270256 10174 raft_consensus.cc:2804] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:02.270695 10174 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:02.271024 10174 raft_consensus.cc:3060] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:02.271626 9711 tablet_replica.cc:333] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:02.272425 9711 raft_consensus.cc:2243] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:02.272902 9711 raft_consensus.cc:2272] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:02.273101 9339 tablet_bootstrap.cc:492] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.273550 9339 ts_tablet_manager.cc:1403] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.007s
I20260926 01:56:02.276122 9339 raft_consensus.cc:359] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.276774 9339 raft_consensus.cc:385] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.277009 9339 raft_consensus.cc:740] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.282759 9339 consensus_queue.cc:260] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.284112 8880 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58226 (request call id 245) took 1293 ms. Trace:
I20260926 01:56:02.284758 9339 ts_tablet_manager.cc:1434] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
W20260926 01:56:02.286307 9003 leader_election.cc:343] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.287726 9711 ts_tablet_manager.cc:1916] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:02.287602 10219 raft_consensus.cc:493] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.288335 10219 raft_consensus.cc:515] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.289119 9339 tablet_bootstrap.cc:492] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:02.290547 10219 leader_election.cc:290] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:02.291857 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18ddaa6f0e748cf930e29cbca3844b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:02.293503 9003 leader_election.cc:343] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.314718 9339 tablet_bootstrap.cc:654] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.316260 8532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 295) took 1066 ms. Trace:
I20260926 01:56:02.316730 8532 rpcz_store.cc:276] 0926 01:56:01.250184 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.265968 (+ 15784us) service_pool.cc:224] Handling call
0926 01:56:01.499574 (+233606us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:01.649790 (+150216us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:02.315502 (+665712us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:02.315864 (+ 362us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:02.316179 (+ 315us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.847897 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.898589 (+ 50692us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.898626 (+ 37us) write_op.cc:435] Acquired schema lock
0926 01:56:01.898643 (+ 17us) tablet.cc:662] Decoding operations
0926 01:56:01.899026 (+ 383us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.899086 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.899115 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:01.899363 (+ 248us) tablet.cc:701] Row locks acquired
0926 01:56:01.899380 (+ 17us) write_op.cc:260] PREPARE: finished
0926 01:56:01.899533 (+ 153us) write_op.cc:270] Start()
0926 01:56:01.899643 (+ 110us) write_op.cc:276] Timestamp: P: 1790387761899450 usec, L: 0
0926 01:56:01.899663 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.912370 (+ 12707us) log.cc:844] Serialized 433 byte log entry
0926 01:56:01.914519 (+ 2149us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c7895e8 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11 00007f717cc1cd3d 00000000005bd4e4 00000000005bd319
0926 01:56:02.298061 (+383542us) op_driver.cc:464] REPLICATION: finished
0926 01:56:02.299148 (+ 1087us) write_op.cc:301] APPLY: starting
0926 01:56:02.299220 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:02.299338 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:02.299360 (+ 22us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:02.299838 (+ 478us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:02.300005 (+ 167us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:02.300151 (+ 146us) write_op.cc:312] APPLY: finished
0926 01:56:02.300399 (+ 248us) log.cc:844] Serialized 29 byte log entry
0926 01:56:02.301232 (+ 833us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:02.301320 (+ 88us) write_op.cc:454] Released schema lock
0926 01:56:02.301362 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:02.301628 (+ 266us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:02.301656 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":893,"apply.run_cpu_time_us":1739,"apply.run_wall_time_us":2439,"num_ops":2,"prepare.queue_time_us":196699,"prepare.run_cpu_time_us":4095,"prepare.run_wall_time_us":71420,"raft.queue_time_us":3017,"raft.run_cpu_time_us":813,"raft.run_wall_time_us":814,"replication_time_us":398277,"spinlock_wait_cycles":27648384}]]}
I20260926 01:56:02.324577 8532 master_service.cc:960] ReplaceTablet: received request to replace tablet b612e8955d5f42dd8658320031fe36ca from {username='slave'} at 127.0.0.1:42048
W20260926 01:56:02.331985 8843 consensus_peers.cc:597] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:02.351810 9339 tablet_bootstrap.cc:492] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.352236 9339 ts_tablet_manager.cc:1403] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.005s
W20260926 01:56:02.352617 8539 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91964074a37441db8b09f78720159586 has no consensus state
I20260926 01:56:02.354764 9339 raft_consensus.cc:359] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.355386 9339 raft_consensus.cc:385] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.355634 9339 raft_consensus.cc:740] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.356441 9339 consensus_queue.cc:260] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.360445 9339 ts_tablet_manager.cc:1434] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260926 01:56:02.361393 9339 tablet_bootstrap.cc:492] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:02.371366 8534 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea75f309937e4ee49b35558117847b05 has no consensus state
I20260926 01:56:02.376713 9339 tablet_bootstrap.cc:654] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.378293 9711 ts_tablet_manager.cc:1929] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:02.392831 9711 log.cc:1199] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/70ef5265eb6244aa96c8195aa30cdfb0
I20260926 01:56:02.393779 9711 ts_tablet_manager.cc:1950] T 70ef5265eb6244aa96c8195aa30cdfb0 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:02.394786 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 176) took 7230 ms. Trace:
W20260926 01:56:02.252056 10211 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.242s user 0.001s sys 0.000s
I20260926 01:56:02.395171 9711 rpcz_store.cc:276] 0926 01:55:55.164020 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.168252 (+ 4232us) service_pool.cc:224] Handling call
0926 01:56:02.271428 (+7103176us) ts_tablet_manager.cc:1151] Deleting tablet 70ef5265eb6244aa96c8195aa30cdfb0
0926 01:56:02.271497 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.340166 (+ 68669us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.378143 (+ 37977us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.394716 (+ 16573us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7102626}
I20260926 01:56:02.397384 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 70ef5265eb6244aa96c8195aa30cdfb0 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]) successfully deleted
I20260926 01:56:02.194654 10135 raft_consensus.cc:2804] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:02.398350 10135 raft_consensus.cc:697] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:02.398733 9711 tablet_replica.cc:333] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:02.399302 10135 consensus_queue.cc:237] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.404484 10143 raft_consensus.cc:493] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.405325 10143 raft_consensus.cc:515] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.407192 9711 raft_consensus.cc:2243] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:02.408703 10143 leader_election.cc:290] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:02.409371 9711 raft_consensus.cc:2272] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:56:02.410864 9003 leader_election.cc:343] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 9bac55da5b164c0f981914d39c87c33e
I20260926 01:56:02.234627 10192 leader_election.cc:290] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:02.414494 9339 tablet_bootstrap.cc:492] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.414944 9339 ts_tablet_manager.cc:1403] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.000s
I20260926 01:56:02.417630 9339 raft_consensus.cc:359] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.418387 9339 raft_consensus.cc:385] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.418673 9339 raft_consensus.cc:740] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.419310 9339 consensus_queue.cc:260] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.423143 9711 ts_tablet_manager.cc:1916] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:02.430773 9339 ts_tablet_manager.cc:1434] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260926 01:56:02.432315 9339 tablet_bootstrap.cc:492] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:02.440814 9340 tablet_bootstrap.cc:492] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:02.441368 9340 ts_tablet_manager.cc:1403] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.631s user 0.038s sys 0.021s
I20260926 01:56:02.443970 9340 raft_consensus.cc:359] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.279738 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17b8b0d54f5645e5963f1de217f94c84" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:02.452582 10143 raft_consensus.cc:493] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.453075 10143 raft_consensus.cc:515] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:02.467470 9196 consensus_meta.cc:287] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.802s user 0.008s sys 0.000s
I20260926 01:56:02.470141 9340 raft_consensus.cc:385] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.451253 10217 tablet.cc:1751] T 28eaf7381f364a2682dd07f2bd90ede3 P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:02.470558 9340 raft_consensus.cc:740] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:56:02.448195 8843 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.474728 10143 leader_election.cc:290] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:02.475706 9340 consensus_queue.cc:260] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.477864 8540 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet bcb000f9234d4382ad019f63015b92e7
I20260926 01:56:02.478056 9339 tablet_bootstrap.cc:654] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.480265 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet bcb000f9234d4382ad019f63015b92e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:02.481590 9340 ts_tablet_manager.cc:1434] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20260926 01:56:02.482540 9340 tablet_bootstrap.cc:492] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:02.487691 9006 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260926 01:56:02.267176 9812 raft_consensus.cc:2243] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:02.516461 9339 tablet_bootstrap.cc:492] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.516933 9339 ts_tablet_manager.cc:1403] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.003s
I20260926 01:56:02.520100 9812 raft_consensus.cc:2272] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:02.522037 9812 ts_tablet_manager.cc:1916] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:02.527029 10174 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.528398 9339 raft_consensus.cc:359] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.529070 9339 raft_consensus.cc:385] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.529317 9339 raft_consensus.cc:740] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.529922 9339 consensus_queue.cc:260] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.542476 9711 ts_tablet_manager.cc:1929] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:02.543145 9711 log.cc:1199] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:56:02.543068 10174 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:02.544015 9711 ts_tablet_manager.cc:1950] T 3bf2fa5fbe4146b5adb27ff09de878d3 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:02.544947 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 177) took 7375 ms. Trace:
I20260926 01:56:02.545887 9340 tablet_bootstrap.cc:654] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.564639 9711 rpcz_store.cc:276] 0926 01:55:55.169259 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.170602 (+ 1343us) service_pool.cc:224] Handling call
0926 01:56:02.398548 (+7227946us) ts_tablet_manager.cc:1151] Deleting tablet 3bf2fa5fbe4146b5adb27ff09de878d3
0926 01:56:02.398614 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.478302 (+ 79688us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.542305 (+ 64003us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.544880 (+ 2575us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7608,"tablet-delete.queue_time_us":7227447}
I20260926 01:56:02.548195 9339 ts_tablet_manager.cc:1434] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20260926 01:56:02.570488 9340 tablet_bootstrap.cc:492] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:02.571030 9340 ts_tablet_manager.cc:1403] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.006s
I20260926 01:56:02.571396 9711 tablet_replica.cc:333] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:02.571444 9339 tablet_bootstrap.cc:492] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:02.571571 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 3bf2fa5fbe4146b5adb27ff09de878d3 (table default.Table_ae4f6e78de7a4f6c8f96a3b179cf7c6d [id=f29287621a4a4cbea49066dde89fb252]) successfully deleted
I20260926 01:56:02.572394 9711 raft_consensus.cc:2243] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:02.572649 10143 raft_consensus.cc:493] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.573256 9711 raft_consensus.cc:2272] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:02.573037 10143 raft_consensus.cc:515] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.573508 9340 raft_consensus.cc:359] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.574146 9340 raft_consensus.cc:385] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.574393 9340 raft_consensus.cc:740] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.575066 10143 leader_election.cc:290] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:02.575201 9340 consensus_queue.cc:260] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.575804 8540 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 296) took 1310 ms. Trace:
I20260926 01:56:02.578176 9339 tablet_bootstrap.cc:654] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:02.579675 10123 lock_manager.cc:585] Waited 2 seconds to obtain row lock on key '<redacted>' (key hash 28713133360477908) tablet 00000000000000000000000000000000 cur holder 0x7b5800040500
I20260926 01:56:02.576668 8540 rpcz_store.cc:276] 0926 01:56:01.265079 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.288640 (+ 23561us) service_pool.cc:224] Handling call
0926 01:56:01.289910 (+ 1270us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:56:01.839301 (+549391us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:56:01.839487 (+ 186us) catalog_manager.cc:2863] Locking tablets
0926 01:56:01.995819 (+156332us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:56:02.477445 (+481626us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:56:02.477674 (+ 229us) catalog_manager.cc:2911] Committing in-memory state
0926 01:56:02.477769 (+ 95us) catalog_manager.cc:2915] Aborting table tasks
0926 01:56:02.575731 (+ 97962us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:01.997982 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:01.998421 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:01.998447 (+ 26us) write_op.cc:435] Acquired schema lock
0926 01:56:01.998479 (+ 32us) tablet.cc:662] Decoding operations
0926 01:56:01.998823 (+ 344us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:01.998865 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:01.998888 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:01.999169 (+ 281us) tablet.cc:701] Row locks acquired
0926 01:56:01.999182 (+ 13us) write_op.cc:260] PREPARE: finished
0926 01:56:01.999311 (+ 129us) write_op.cc:270] Start()
0926 01:56:01.999399 (+ 88us) write_op.cc:276] Timestamp: P: 1790387761999227 usec, L: 0
0926 01:56:01.999415 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:56:01.999975 (+ 560us) log.cc:844] Serialized 759 byte log entry
0926 01:56:02.360243 (+360268us) op_driver.cc:464] REPLICATION: finished
0926 01:56:02.364000 (+ 3757us) write_op.cc:301] APPLY: starting
0926 01:56:02.364062 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:02.364171 (+ 109us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:02.364298 (+ 127us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:02.364593 (+ 295us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:02.364741 (+ 148us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:56:02.364784 (+ 43us) write_op.cc:312] APPLY: finished
0926 01:56:02.364974 (+ 190us) log.cc:844] Serialized 29 byte log entry
0926 01:56:02.365135 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:02.365210 (+ 75us) write_op.cc:454] Released schema lock
0926 01:56:02.365243 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:02.365469 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:02.365494 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3557,"apply.run_cpu_time_us":1474,"apply.run_wall_time_us":1475,"num_ops":2,"prepare.queue_time_us":128,"prepare.run_cpu_time_us":2251,"prepare.run_wall_time_us":2250,"replication_time_us":360722}]]}
I20260926 01:56:02.580854 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet bcb000f9234d4382ad019f63015b92e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:02.418284 9341 ts_tablet_manager.cc:1403] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 1.368s user 0.018s sys 0.025s
I20260926 01:56:02.533533 10186 ts_tablet_manager.cc:1929] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:02.586485 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_7d32da88077d4060913cc8ae86df83f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:02.461539 10127 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.000s sys 0.012s
W20260926 01:56:02.587443 10127 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.000s sys 0.015s
W20260926 01:56:02.588198 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d32da88077d4060913cc8ae86df83f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:02.589757 9711 ts_tablet_manager.cc:1916] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:02.592761 8534 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_86c054848f05455c849277b21ead1bd7" } new_table_name: "default.Table_aa6bb98036044d52970008af2616d33c" modify_external_catalogs: true
I20260926 01:56:02.606312 8536 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_d908977ed46746238f564a8c31a4c83e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:02.608497 8536 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d908977ed46746238f564a8c31a4c83e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:02.611939 10186 log.cc:1199] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/3571967749844fd7859cf179a93cf403
I20260926 01:56:02.616701 10186 ts_tablet_manager.cc:1950] T 3571967749844fd7859cf179a93cf403 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:02.615103 9190 tablet_service.cc:1558] Processing DeleteTablet for tablet bcb000f9234d4382ad019f63015b92e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:02.627734 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 266) took 1578 ms. Trace:
I20260926 01:56:02.629040 10186 rpcz_store.cc:276] 0926 01:56:01.048886 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.049017 (+ 131us) service_pool.cc:224] Handling call
0926 01:56:01.293324 (+244307us) ts_tablet_manager.cc:1151] Deleting tablet 3571967749844fd7859cf179a93cf403
0926 01:56:01.611765 (+318441us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.115159 (+503394us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.533384 (+418225us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.627662 (+ 94278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33415,"tablet-delete.queue_time_us":243652,"thread_start_us":328238,"threads_started":1}
I20260926 01:56:02.635011 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 3571967749844fd7859cf179a93cf403 (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09]) successfully deleted
I20260926 01:56:02.636955 9341 raft_consensus.cc:359] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.637957 9341 raft_consensus.cc:385] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.638279 9341 raft_consensus.cc:740] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:02.639014 9341 consensus_queue.cc:260] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:02.641052 9341 ts_tablet_manager.cc:1434] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20260926 01:56:02.642659 9341 tablet_bootstrap.cc:492] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:02.648010 9339 tablet_bootstrap.cc:492] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.648479 9339 ts_tablet_manager.cc:1403] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.000s
I20260926 01:56:02.646652 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 268) took 1480 ms. Trace:
I20260926 01:56:02.650652 10186 rpcz_store.cc:276] 0926 01:56:01.166268 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.260667 (+ 94399us) service_pool.cc:224] Handling call
0926 01:56:02.630617 (+1369950us) ts_tablet_manager.cc:1151] Deleting tablet 3bf2fa5fbe4146b5adb27ff09de878d3
0926 01:56:02.630667 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.646583 (+ 15916us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361781}
I20260926 01:56:02.651297 9339 raft_consensus.cc:359] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.652045 9339 raft_consensus.cc:385] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:02.652314 9339 raft_consensus.cc:740] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
W20260926 01:56:02.652782 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 because the tablet was not found. No further retry: Not found: Tablet not found: 3bf2fa5fbe4146b5adb27ff09de878d3
I20260926 01:56:02.652948 9339 consensus_queue.cc:260] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:02.654160 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 269) took 1032 ms. Trace:
I20260926 01:56:02.654539 10186 rpcz_store.cc:276] 0926 01:56:01.621831 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.621945 (+ 114us) service_pool.cc:224] Handling call
0926 01:56:02.653852 (+1031907us) ts_tablet_manager.cc:1151] Deleting tablet ff004cf835c34095a836ad683cec9256
0926 01:56:02.653901 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.654100 (+ 199us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031299}
W20260926 01:56:02.655967 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet ff004cf835c34095a836ad683cec9256 because the tablet was not found. No further retry: Not found: Tablet not found: ff004cf835c34095a836ad683cec9256
I20260926 01:56:02.659548 9339 ts_tablet_manager.cc:1434] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260926 01:56:02.660187 9711 ts_tablet_manager.cc:1929] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:02.660501 9339 tablet_bootstrap.cc:492] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:02.660593 9711 log.cc:1199] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/c36ea09cf15647c0a344c9ba9b289d61
I20260926 01:56:02.661420 9711 ts_tablet_manager.cc:1950] T c36ea09cf15647c0a344c9ba9b289d61 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:02.663959 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 271) took 1039 ms. Trace:
I20260926 01:56:02.664381 10186 rpcz_store.cc:276] 0926 01:56:01.624145 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.624239 (+ 94us) service_pool.cc:224] Handling call
0926 01:56:02.663605 (+1039366us) ts_tablet_manager.cc:1151] Deleting tablet 72f666b1c069435b8ff0a3fae9f1e873
0926 01:56:02.663657 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.663894 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038824}
W20260926 01:56:02.667224 8527 catalog_manager.cc:5030] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 72f666b1c069435b8ff0a3fae9f1e873 because the tablet was not found. No further retry: Not found: Tablet not found: 72f666b1c069435b8ff0a3fae9f1e873
I20260926 01:56:02.673112 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 178) took 7501 ms. Trace:
I20260926 01:56:02.673434 9711 rpcz_store.cc:276] 0926 01:55:55.171468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.174488 (+ 3020us) service_pool.cc:224] Handling call
0926 01:56:02.571087 (+7396599us) ts_tablet_manager.cc:1151] Deleting tablet c36ea09cf15647c0a344c9ba9b289d61
0926 01:56:02.571173 (+ 86us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.635756 (+ 64583us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.655676 (+ 19920us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.673031 (+ 17355us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7396076}
I20260926 01:56:02.675001 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet c36ea09cf15647c0a344c9ba9b289d61 (table default.Table_722ce830650d49ffa0b00028e056e1db [id=b10f8555970c4add9293d27073a1175b]) successfully deleted
I20260926 01:56:02.682876 9711 tablet_replica.cc:333] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:02.683722 9711 raft_consensus.cc:2243] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:02.684243 9711 raft_consensus.cc:2272] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:02.685756 9339 tablet_bootstrap.cc:654] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.686198 9711 ts_tablet_manager.cc:1916] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:02.284529 8880 rpcz_store.cc:276] 0926 01:56:00.990675 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.990810 (+ 135us) service_pool.cc:224] Handling call
0926 01:56:01.022565 (+ 31755us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0926 01:56:01.022670 (+ 105us) ts_tablet_manager.cc:682] Creating new metadata...
0926 01:56:01.413745 (+391075us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:01.908203 (+494458us) tablet_replica.cc:206] Creating consensus instance
0926 01:56:02.284046 (+375843us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:02.725718 9340 ts_tablet_manager.cc:1434] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.154s user 0.006s sys 0.000s
I20260926 01:56:02.721987 8533 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_05db3de2a0574423980de7d7b7098324"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:02.729212 8533 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05db3de2a0574423980de7d7b7098324 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:02.733470 9340 tablet_bootstrap.cc:492] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:02.739097 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f49a5a10ea4f7a821d89f18e3af560" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:02.795153 10186 tablet_replica.cc:333] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:02.796123 10186 raft_consensus.cc:2243] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:02.796710 10186 raft_consensus.cc:2272] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:02.810986 9340 tablet_bootstrap.cc:654] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.821449 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fdb7a7328c54031bd3afac1636d0f75" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:02.825258 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17b8b0d54f5645e5963f1de217f94c84" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:02.844466 8843 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.859886 10239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:02.860608 10239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:02.867144 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c448caeeedfb4fc88eee88e40eebfeeb" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:02.868096 9212 raft_consensus.cc:2468] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:02.872821 10239 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:02.872785 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 338 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:02.874497 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 338 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:02.878930 8540 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_4d73c5f209a7403bb51e1ef3ce6084bb" } new_table_name: "default.Table_57ad2cd7ce1e4bc6a071b8a0b985c3f2" modify_external_catalogs: true
I20260926 01:56:02.881415 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:02.882189 10239 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260926 01:56:02.919998 10219 raft_consensus.cc:493] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:02.920502 10219 raft_consensus.cc:515] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:02.922755 9003 leader_election.cc:343] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:02.923638 9003 leader_election.cc:343] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.931769 9711 ts_tablet_manager.cc:1929] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:02.932190 9711 log.cc:1199] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/72f666b1c069435b8ff0a3fae9f1e873
I20260926 01:56:02.932849 9711 ts_tablet_manager.cc:1950] T 72f666b1c069435b8ff0a3fae9f1e873 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:02.933303 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:02.933657 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 179) took 7758 ms. Trace:
I20260926 01:56:02.933924 9711 rpcz_store.cc:276] 0926 01:55:55.175437 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.350865 (+175428us) service_pool.cc:224] Handling call
0926 01:56:02.682678 (+7331813us) ts_tablet_manager.cc:1151] Deleting tablet 72f666b1c069435b8ff0a3fae9f1e873
0926 01:56:02.682748 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.748090 (+ 65342us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.931614 (+183524us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.933603 (+ 1989us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7331169}
I20260926 01:56:02.944128 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 72f666b1c069435b8ff0a3fae9f1e873 (table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb]) successfully deleted
I20260926 01:56:02.946921 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17b8b0d54f5645e5963f1de217f94c84" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:02.949589 9070 raft_consensus.cc:3060] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:02.951418 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4623456312284a698ae69d4054909647" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:02.954756 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18ddaa6f0e748cf930e29cbca3844b1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:02.939967 10219 leader_election.cc:290] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:02.993537 9339 tablet_bootstrap.cc:492] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:02.994050 9339 ts_tablet_manager.cc:1403] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.334s user 0.061s sys 0.051s
W20260926 01:56:02.994524 9006 leader_election.cc:343] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: ff004cf835c34095a836ad683cec9256
I20260926 01:56:02.994930 9006 leader_election.cc:304] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:56:02.997114 9003 leader_election.cc:343] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:02.996291 10219 raft_consensus.cc:2749] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:03.004333 9339 raft_consensus.cc:359] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.005422 9339 raft_consensus.cc:385] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.006105 9339 raft_consensus.cc:740] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:03.006927 9339 consensus_queue.cc:260] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.008999 9339 ts_tablet_manager.cc:1434] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
W20260926 01:56:03.011605 8525 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet ea75f309937e4ee49b35558117847b05 (table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ea75f309937e4ee49b35558117847b05 has no consensus state
I20260926 01:56:03.012436 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 183) took 7631 ms. Trace:
I20260926 01:56:03.024530 10219 consensus_queue.cc:1048] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:03.032456 9006 leader_election.cc:304] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
W20260926 01:56:03.036082 9006 leader_election.cc:343] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.036473 9006 leader_election.cc:304] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.037125 10143 raft_consensus.cc:2804] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:03.037441 10143 raft_consensus.cc:493] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:03.037747 10143 raft_consensus.cc:3060] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:03.037922 9006 leader_election.cc:343] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 9bac55da5b164c0f981914d39c87c33e
I20260926 01:56:03.038781 10127 raft_consensus.cc:2749] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:03.040534 9711 rpcz_store.cc:276] 0926 01:55:55.380914 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.381068 (+ 154us) service_pool.cc:224] Handling call
0926 01:56:02.945218 (+7564150us) ts_tablet_manager.cc:1151] Deleting tablet 0d068e93fd374a6091d7473db069a7fa
0926 01:56:02.973263 (+ 28045us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.012369 (+ 39106us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7563547}
I20260926 01:56:03.038270 9006 leader_election.cc:304] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.042129 10130 raft_consensus.cc:2749] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:03.044198 8527 catalog_manager.cc:5043] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 0d068e93fd374a6091d7473db069a7fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d068e93fd374a6091d7473db069a7fa already in progress: creating tablet
I20260926 01:56:02.969542 9341 tablet_bootstrap.cc:654] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:02.975970 10206 tablet.cc:1751] T f2ee16b0c3e84936abba8248f194786a P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:02.981453 10186 ts_tablet_manager.cc:1916] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:02.986527 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fdb7a7328c54031bd3afac1636d0f75" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:02.987937 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f49a5a10ea4f7a821d89f18e3af560" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.007309 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.059504 9070 raft_consensus.cc:2468] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:03.072045 10227 consensus_queue.cc:1048] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260926 01:56:03.074373 10174 raft_consensus.cc:493] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.074905 10174 raft_consensus.cc:515] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.076951 10174 leader_election.cc:290] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:03.071300 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:03.078855 8905 raft_consensus.cc:3060] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:03.061509 10143 raft_consensus.cc:515] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.064791 9339 tablet_bootstrap.cc:492] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:03.094424 10143 leader_election.cc:290] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:03.065397 9711 tablet_replica.cc:333] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:03.100699 10143 raft_consensus.cc:493] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.100987 9711 raft_consensus.cc:2243] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:03.101470 9711 raft_consensus.cc:2272] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:03.101191 10143 raft_consensus.cc:515] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.103183 10143 leader_election.cc:290] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:03.103706 9711 ts_tablet_manager.cc:1916] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:03.094532 10174 raft_consensus.cc:493] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.116233 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c448caeeedfb4fc88eee88e40eebfeeb" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:03.119673 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e12827ad93c47cf9013e238215ceb7a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:03.121230 9003 leader_election.cc:343] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.121206 9339 tablet_bootstrap.cc:654] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:03.121325 10174 raft_consensus.cc:515] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.123248 8907 raft_consensus.cc:3060] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:03.127183 10174 leader_election.cc:290] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:03.136124 9812 ts_tablet_manager.cc:1929] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:03.136689 9812 log.cc:1199] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/72f666b1c069435b8ff0a3fae9f1e873
I20260926 01:56:03.144755 9339 tablet_bootstrap.cc:492] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:03.145226 9339 ts_tablet_manager.cc:1403] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.000s
I20260926 01:56:03.151546 9340 tablet_bootstrap.cc:492] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:03.152035 9340 ts_tablet_manager.cc:1403] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.419s user 0.018s sys 0.002s
I20260926 01:56:03.157347 10191 consensus_queue.cc:1048] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:03.159380 8905 raft_consensus.cc:2468] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:03.161379 9812 ts_tablet_manager.cc:1950] T 72f666b1c069435b8ff0a3fae9f1e873 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:03.165287 8907 raft_consensus.cc:2468] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
W20260926 01:56:03.178558 9006 leader_election.cc:343] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.179032 9006 leader_election.cc:304] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.180092 10143 raft_consensus.cc:2749] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:03.180078 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 109480us
I20260926 01:56:03.182807 9341 tablet_bootstrap.cc:492] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:03.182782 8843 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.183235 9341 ts_tablet_manager.cc:1403] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.541s user 0.014s sys 0.000s
I20260926 01:56:03.183832 10227 raft_consensus.cc:2804] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:03.184351 10227 raft_consensus.cc:697] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:03.185225 10227 consensus_queue.cc:237] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.186082 9341 raft_consensus.cc:359] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.186736 9341 raft_consensus.cc:385] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.186992 9341 raft_consensus.cc:740] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:56:03.197470 9006 leader_election.cc:343] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.197969 9006 leader_election.cc:304] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.199023 10143 raft_consensus.cc:2749] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:03.201447 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 252) took 2037 ms. Trace:
W20260926 01:56:03.201555 9006 leader_election.cc:343] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.201917 9006 leader_election.cc:304] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.202558 10143 raft_consensus.cc:2749] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:03.204815 9006 leader_election.cc:343] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.205169 9006 leader_election.cc:304] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.204874 9812 rpcz_store.cc:276] 0926 01:56:01.164395 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.164482 (+ 87us) service_pool.cc:224] Handling call
0926 01:56:02.266210 (+1101728us) ts_tablet_manager.cc:1151] Deleting tablet 72f666b1c069435b8ff0a3fae9f1e873
0926 01:56:02.266248 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:02.724703 (+458455us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:02.840186 (+115483us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.201341 (+361155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7954,"tablet-delete.queue_time_us":1101275}
I20260926 01:56:03.205797 10143 raft_consensus.cc:2749] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:03.206429 9812 tablet_replica.cc:333] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:03.211900 9812 raft_consensus.cc:2243] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:03.212431 9812 raft_consensus.cc:2272] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:03.214551 9812 ts_tablet_manager.cc:1916] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:03.216355 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:03.217136 9072 raft_consensus.cc:2393] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:56:03.217203 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "661148692f9e4127af9abf5cc6cc0f54" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.223330 9003 leader_election.cc:304] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:03.224100 10143 raft_consensus.cc:2804] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:03.224848 10143 raft_consensus.cc:697] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:03.225683 10143 consensus_queue.cc:237] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.247323 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c448caeeedfb4fc88eee88e40eebfeeb" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.248044 9217 raft_consensus.cc:2468] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:03.258160 9812 ts_tablet_manager.cc:1929] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:03.265455 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c448caeeedfb4fc88eee88e40eebfeeb" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:03.258618 9812 log.cc:1199] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:56:03.267477 9812 ts_tablet_manager.cc:1950] T f100eb5dc9a8452e943d4cce5284bc8a P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:03.269783 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 253) took 2104 ms. Trace:
I20260926 01:56:03.275377 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 72f666b1c069435b8ff0a3fae9f1e873 (table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb]) successfully deleted
I20260926 01:56:03.270105 9812 rpcz_store.cc:276] 0926 01:56:01.165212 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.165291 (+ 79us) service_pool.cc:224] Handling call
0926 01:56:03.206224 (+2040933us) ts_tablet_manager.cc:1151] Deleting tablet f100eb5dc9a8452e943d4cce5284bc8a
0926 01:56:03.206292 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.239072 (+ 32780us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.258009 (+ 18937us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.269690 (+ 11681us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2040477}
I20260926 01:56:03.278892 9812 tablet_replica.cc:333] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:03.279769 9812 raft_consensus.cc:2243] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:03.280294 9812 raft_consensus.cc:2272] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:03.280593 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c448caeeedfb4fc88eee88e40eebfeeb" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:03.281157 9217 raft_consensus.cc:3060] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:03.282541 9812 ts_tablet_manager.cc:1916] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:03.286803 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e12827ad93c47cf9013e238215ceb7a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.290438 9217 raft_consensus.cc:2468] T c448caeeedfb4fc88eee88e40eebfeeb P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:03.213968 9340 raft_consensus.cc:359] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.298882 9340 raft_consensus.cc:385] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.299461 9340 raft_consensus.cc:740] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:03.300271 9340 consensus_queue.cc:260] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.302956 9340 ts_tablet_manager.cc:1434] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.151s user 0.008s sys 0.000s
I20260926 01:56:03.304008 9340 tablet_bootstrap.cc:492] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:03.305109 8843 leader_election.cc:304] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:03.322789 10219 raft_consensus.cc:493] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.323336 10219 raft_consensus.cc:515] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.325480 10219 leader_election.cc:290] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:03.326191 9339 raft_consensus.cc:359] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.336710 9339 raft_consensus.cc:385] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.337122 9339 raft_consensus.cc:740] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
W20260926 01:56:03.342020 9006 leader_election.cc:343] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.342705 9006 leader_election.cc:304] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.344172 10130 raft_consensus.cc:2749] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:03.349620 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet b612e8955d5f42dd8658320031fe36ca
I20260926 01:56:03.351209 8532 catalog_manager.cc:6934] ReplaceTablet: tablet b612e8955d5f42dd8658320031fe36ca deleted and replaced by tablet 1279a644e6604fb0bc20f53cd23dd8be
I20260926 01:56:03.351578 8532 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 304) took 1201 ms. Trace:
I20260926 01:56:03.378737 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet b612e8955d5f42dd8658320031fe36ca with delete_type TABLET_DATA_DELETED (replaced by tablet 1279a644e6604fb0bc20f53cd23dd8be) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:03.379702 10219 raft_consensus.cc:493] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.380213 10219 raft_consensus.cc:515] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.351848 8532 rpcz_store.cc:276] 0926 01:56:02.150182 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:02.324287 (+174105us) service_pool.cc:224] Handling call
0926 01:56:03.351500 (+1027213us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:02.530691 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:02.531146 (+ 455us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:02.531171 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:56:02.531184 (+ 13us) tablet.cc:662] Decoding operations
0926 01:56:02.531513 (+ 329us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:02.531558 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:02.531581 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:02.531792 (+ 211us) tablet.cc:701] Row locks acquired
0926 01:56:02.531806 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:56:02.531931 (+ 125us) write_op.cc:270] Start()
0926 01:56:02.532015 (+ 84us) write_op.cc:276] Timestamp: P: 1790387762531855 usec, L: 0
0926 01:56:02.532031 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:56:02.584863 (+ 52832us) log.cc:844] Serialized 625 byte log entry
0926 01:56:02.585607 (+ 744us) spinlock_profiling.cc:243] Waited 52.2 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c7895e8 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11 00007f717cc1cd3d 00000000005bd4e4 00000000005bd319
0926 01:56:03.235725 (+650118us) op_driver.cc:464] REPLICATION: finished
0926 01:56:03.292680 (+ 56955us) write_op.cc:301] APPLY: starting
0926 01:56:03.292751 (+ 71us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:03.292874 (+ 123us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:03.292897 (+ 23us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:03.293313 (+ 416us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:03.293576 (+ 263us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:03.293626 (+ 50us) write_op.cc:312] APPLY: finished
0926 01:56:03.293839 (+ 213us) log.cc:844] Serialized 29 byte log entry
0926 01:56:03.318093 (+ 24254us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:03.348899 (+ 30806us) write_op.cc:454] Released schema lock
0926 01:56:03.348938 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56733,"num_ops":2,"prepare.queue_time_us":202768,"prepare.run_cpu_time_us":3419,"prepare.run_wall_time_us":55046,"replication_time_us":703558,"spinlock_wait_cycles":120094080,"thread_start_us":58711,"threads_started":1}]]}
I20260926 01:56:03.382285 10219 leader_election.cc:290] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:03.307068 9341 consensus_queue.cc:260] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.329404 10191 consensus_queue.cc:1048] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:03.356101 9187 tablet_service.cc:1558] Processing DeleteTablet for tablet b612e8955d5f42dd8658320031fe36ca with delete_type TABLET_DATA_DELETED (replaced by tablet 1279a644e6604fb0bc20f53cd23dd8be) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:03.316630 9812 ts_tablet_manager.cc:1929] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:03.359452 9265 heartbeater.cc:499] Master 127.8.76.188:38143 was elected leader, sending a full tablet report...
I20260926 01:56:03.416257 9812 log.cc:1199] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/943a990fa10948e8961c02e9999569f5
W20260926 01:56:03.416528 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 111482us
I20260926 01:56:03.417110 9812 ts_tablet_manager.cc:1950] T 943a990fa10948e8961c02e9999569f5 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:03.417649 10227 raft_consensus.cc:2749] T 661148692f9e4127af9abf5cc6cc0f54 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:03.419431 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 254) took 2253 ms. Trace:
I20260926 01:56:03.419756 9812 rpcz_store.cc:276] 0926 01:56:01.165958 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.166044 (+ 86us) service_pool.cc:224] Handling call
0926 01:56:03.278669 (+2112625us) ts_tablet_manager.cc:1151] Deleting tablet 943a990fa10948e8961c02e9999569f5
0926 01:56:03.278747 (+ 78us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.299064 (+ 20317us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.316494 (+ 17430us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.419368 (+102874us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2112123}
I20260926 01:56:03.421025 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 257) took 2229 ms. Trace:
I20260926 01:56:03.421315 9812 rpcz_store.cc:276] 0926 01:56:01.191171 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.191299 (+ 128us) service_pool.cc:224] Handling call
0926 01:56:03.420701 (+2229402us) ts_tablet_manager.cc:1151] Deleting tablet c27f1dd7bb0449a6a3fa371a8636e09b
0926 01:56:03.420762 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.420965 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228052}
I20260926 01:56:03.422580 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 256) took 2231 ms. Trace:
I20260926 01:56:03.422859 9812 rpcz_store.cc:276] 0926 01:56:01.190873 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.191004 (+ 131us) service_pool.cc:224] Handling call
0926 01:56:03.422265 (+2231261us) ts_tablet_manager.cc:1151] Deleting tablet b659eca3c97144aaacd41e747aecd96d
0926 01:56:03.422315 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.422521 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2229141}
I20260926 01:56:03.424080 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 258) took 2232 ms. Trace:
I20260926 01:56:03.424363 9812 rpcz_store.cc:276] 0926 01:56:01.191461 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.196656 (+ 5195us) service_pool.cc:224] Handling call
0926 01:56:03.423782 (+2227126us) ts_tablet_manager.cc:1151] Deleting tablet 39cd9130640d4b14beef42e9f1b36b99
0926 01:56:03.423834 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.424025 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2226427}
I20260926 01:56:03.425400 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 259) took 2233 ms. Trace:
I20260926 01:56:03.425693 9812 rpcz_store.cc:276] 0926 01:56:01.191741 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.211298 (+ 19557us) service_pool.cc:224] Handling call
0926 01:56:03.425141 (+2213843us) ts_tablet_manager.cc:1151] Deleting tablet 1cba678b66744dcfb4d13ae07ebec33f
0926 01:56:03.425187 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.425353 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213146}
I20260926 01:56:03.332983 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d56f3a5dfed4ac78f1da92e22db558a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:03.401530 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:03.427353 8906 raft_consensus.cc:2468] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
W20260926 01:56:03.428192 9003 leader_election.cc:343] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.428893 9003 leader_election.cc:304] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:03.395970 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d56f3a5dfed4ac78f1da92e22db558a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.437182 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 260) took 2224 ms. Trace:
I20260926 01:56:03.437618 9812 rpcz_store.cc:276] 0926 01:56:01.212451 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.218227 (+ 5776us) service_pool.cc:224] Handling call
0926 01:56:03.436837 (+2218610us) ts_tablet_manager.cc:1151] Deleting tablet 70ef5265eb6244aa96c8195aa30cdfb0
0926 01:56:03.436904 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.437118 (+ 214us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2207558}
I20260926 01:56:03.438944 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 261) took 2219 ms. Trace:
I20260926 01:56:03.439369 9812 rpcz_store.cc:276] 0926 01:56:01.219353 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.219445 (+ 92us) service_pool.cc:224] Handling call
0926 01:56:03.438611 (+2219166us) ts_tablet_manager.cc:1151] Deleting tablet 9bac55da5b164c0f981914d39c87c33e
0926 01:56:03.438667 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.438890 (+ 223us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2218675}
I20260926 01:56:03.440383 9812 tablet_replica.cc:333] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:03.441202 9812 raft_consensus.cc:2243] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:03.442108 9812 raft_consensus.cc:2272] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:03.444275 9812 ts_tablet_manager.cc:1916] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:03.444844 9341 ts_tablet_manager.cc:1434] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.261s user 0.007s sys 0.000s
I20260926 01:56:03.447389 10219 raft_consensus.cc:2804] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:56:03.447796 10219 raft_consensus.cc:493] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:03.448189 10219 raft_consensus.cc:3060] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:03.337927 9339 consensus_queue.cc:260] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.417230 8843 leader_election.cc:304] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
W20260926 01:56:03.475324 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 58176.4us
I20260926 01:56:03.476316 10174 raft_consensus.cc:2764] T c448caeeedfb4fc88eee88e40eebfeeb P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20260926 01:56:03.476593 9006 leader_election.cc:343] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.477236 9006 leader_election.cc:304] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:56:03.481419 8534 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 consensus state has no leader
I20260926 01:56:03.487367 9341 tablet_bootstrap.cc:492] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:03.491408 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 17b8b0d54f5645e5963f1de217f94c84 from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:03.501446 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet f100eb5dc9a8452e943d4cce5284bc8a (table default.Table_37930b35bf904b5aabec0d568896d39b [id=b6747abf32974ff4a3bf5da14e923b1e]) successfully deleted
I20260926 01:56:03.502650 10127 raft_consensus.cc:2749] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:03.505306 9217 raft_consensus.cc:3060] T 56b0e40e5e964b90bd2dd158bbc4497a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:03.519820 9339 ts_tablet_manager.cc:1434] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.374s user 0.008s sys 0.000s
I20260926 01:56:03.527026 8534 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_82723302583843e2b50ce4a3d0e4adcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:56:03.536106 8536 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_129e4100f7f245e9877924fd11902aea" } modify_external_catalogs: true
I20260926 01:56:03.536717 8536 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_129e4100f7f245e9877924fd11902aea" } modify_external_catalogs: true
I20260926 01:56:03.539394 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.540005 9218 raft_consensus.cc:2468] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:03.558125 9812 ts_tablet_manager.cc:1929] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:03.558617 9812 log.cc:1199] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/cbf8818e65d74a32ba21dcb8a9a56a62
I20260926 01:56:03.559417 9812 ts_tablet_manager.cc:1950] T cbf8818e65d74a32ba21dcb8a9a56a62 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:03.561736 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 265) took 2177 ms. Trace:
I20260926 01:56:03.562047 9812 rpcz_store.cc:276] 0926 01:56:01.384810 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.384922 (+ 112us) service_pool.cc:224] Handling call
0926 01:56:03.440224 (+2055302us) ts_tablet_manager.cc:1151] Deleting tablet cbf8818e65d74a32ba21dcb8a9a56a62
0926 01:56:03.440279 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.537866 (+ 97587us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.557970 (+ 20104us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.561633 (+ 3663us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2054717}
I20260926 01:56:03.563484 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 266) took 2177 ms. Trace:
I20260926 01:56:03.563809 9812 rpcz_store.cc:276] 0926 01:56:01.386018 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.390701 (+ 4683us) service_pool.cc:224] Handling call
0926 01:56:03.563060 (+2172359us) ts_tablet_manager.cc:1151] Deleting tablet 8d198e51b88948548cfd95dd893a26ef
0926 01:56:03.563113 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.563433 (+ 320us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171711}
I20260926 01:56:03.564752 9812 tablet_replica.cc:333] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:03.565501 9812 raft_consensus.cc:2243] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:03.570595 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 943a990fa10948e8961c02e9999569f5 (table default.Table_3907f5fafff14aec94d0a9e331e86dc9 [id=cc59eec581574b65a6332863423dab90]) successfully deleted
I20260926 01:56:03.571523 9812 raft_consensus.cc:2272] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:56:03.573571 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet c27f1dd7bb0449a6a3fa371a8636e09b because the tablet was not found. No further retry: Not found: Tablet not found: c27f1dd7bb0449a6a3fa371a8636e09b
I20260926 01:56:03.573787 9812 ts_tablet_manager.cc:1916] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:03.575160 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet b659eca3c97144aaacd41e747aecd96d because the tablet was not found. No further retry: Not found: Tablet not found: b659eca3c97144aaacd41e747aecd96d
W20260926 01:56:03.576506 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 39cd9130640d4b14beef42e9f1b36b99 because the tablet was not found. No further retry: Not found: Tablet not found: 39cd9130640d4b14beef42e9f1b36b99
W20260926 01:56:03.577862 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 1cba678b66744dcfb4d13ae07ebec33f because the tablet was not found. No further retry: Not found: Tablet not found: 1cba678b66744dcfb4d13ae07ebec33f
I20260926 01:56:03.581681 9341 tablet_bootstrap.cc:654] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:03.609359 10186 ts_tablet_manager.cc:1929] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:03.610000 10186 log.cc:1199] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:56:03.610900 10186 ts_tablet_manager.cc:1950] T f100eb5dc9a8452e943d4cce5284bc8a P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:56:03.613082 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 70ef5265eb6244aa96c8195aa30cdfb0 because the tablet was not found. No further retry: Not found: Tablet not found: 70ef5265eb6244aa96c8195aa30cdfb0
W20260926 01:56:03.613873 8534 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82723302583843e2b50ce4a3d0e4adcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:56:03.618396 8524 catalog_manager.cc:5030] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 9bac55da5b164c0f981914d39c87c33e because the tablet was not found. No further retry: Not found: Tablet not found: 9bac55da5b164c0f981914d39c87c33e
I20260926 01:56:03.625083 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet b612e8955d5f42dd8658320031fe36ca with delete_type TABLET_DATA_DELETED (replaced by tablet 1279a644e6604fb0bc20f53cd23dd8be) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:03.625088 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet cbf8818e65d74a32ba21dcb8a9a56a62 (table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]) successfully deleted
I20260926 01:56:03.629827 9812 ts_tablet_manager.cc:1929] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:03.630338 9812 log.cc:1199] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:03.630486 10191 consensus_queue.cc:1048] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:03.631090 9812 ts_tablet_manager.cc:1950] T 1f6d9fedc7b241b59b25d591c7dc2fea P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:03.632030 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 267) took 2233 ms. Trace:
I20260926 01:56:03.632333 9812 rpcz_store.cc:276] 0926 01:56:01.398766 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.450757 (+ 51991us) service_pool.cc:224] Handling call
0926 01:56:03.564583 (+2113826us) ts_tablet_manager.cc:1151] Deleting tablet 1f6d9fedc7b241b59b25d591c7dc2fea
0926 01:56:03.564646 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.581583 (+ 16937us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.629677 (+ 48094us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.631957 (+ 2280us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2113136}
I20260926 01:56:03.633839 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 268) took 1153 ms. Trace:
I20260926 01:56:03.634147 9812 rpcz_store.cc:276] 0926 01:56:02.480014 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:02.480124 (+ 110us) service_pool.cc:224] Handling call
0926 01:56:03.633423 (+1153299us) ts_tablet_manager.cc:1151] Deleting tablet bcb000f9234d4382ad019f63015b92e7
0926 01:56:03.633475 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.633787 (+ 312us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152716}
I20260926 01:56:03.529840 10219 raft_consensus.cc:515] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:56:03.639664 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 8d198e51b88948548cfd95dd893a26ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d198e51b88948548cfd95dd893a26ef already in progress: creating tablet
I20260926 01:56:03.566636 9339 tablet_bootstrap.cc:492] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:03.649572 10127 raft_consensus.cc:493] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.650087 10127 raft_consensus.cc:515] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.653146 8524 catalog_manager.cc:5063] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): tablet 1f6d9fedc7b241b59b25d591c7dc2fea (table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]) successfully deleted
I20260926 01:56:03.653381 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f10cb68586b4a73abb86641b90c12f7" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:03.654881 10127 leader_election.cc:290] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:03.655112 9003 leader_election.cc:343] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.674870 10219 leader_election.cc:290] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:03.676782 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:03.677376 8906 raft_consensus.cc:3060] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:56:03.678018 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet bcb000f9234d4382ad019f63015b92e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcb000f9234d4382ad019f63015b92e7 already in progress: creating tablet
W20260926 01:56:03.679260 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet b612e8955d5f42dd8658320031fe36ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b612e8955d5f42dd8658320031fe36ca already in progress: creating tablet
I20260926 01:56:03.686152 8906 raft_consensus.cc:2468] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 2.
I20260926 01:56:03.687479 9003 leader_election.cc:304] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:03.688380 10219 raft_consensus.cc:2804] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:56:03.689464 9340 tablet_bootstrap.cc:654] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:03.691924 10143 raft_consensus.cc:493] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 01:56:03.692473 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 59956.8us
I20260926 01:56:03.692490 10143 raft_consensus.cc:515] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.696127 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f10cb68586b4a73abb86641b90c12f7" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.698925 10143 leader_election.cc:290] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:03.700255 10262 raft_consensus.cc:493] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.700757 10262 raft_consensus.cc:515] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.704768 10262 leader_election.cc:290] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:03.705567 9003 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:56:03.707676 10219 raft_consensus.cc:697] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [term 2 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:03.708324 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:03.708683 10219 consensus_queue.cc:237] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:03.709046 9072 raft_consensus.cc:2468] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:03.713958 8843 leader_election.cc:304] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:03.715456 10262 raft_consensus.cc:2804] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:03.715873 10262 raft_consensus.cc:493] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:03.716246 10262 raft_consensus.cc:3060] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:03.620384 9711 ts_tablet_manager.cc:1929] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:03.739974 9711 log.cc:1199] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:03.740854 9711 ts_tablet_manager.cc:1950] T 17b8b0d54f5645e5963f1de217f94c84 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:03.741902 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 182) took 8375 ms. Trace:
W20260926 01:56:03.759630 10241 log.cc:927] Time spent T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Append to log took a long time: real 0.683s user 0.022s sys 0.008s
I20260926 01:56:03.773490 9339 tablet_bootstrap.cc:700] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2: Time spent opening tablet: real 0.111s user 0.004s sys 0.000s
I20260926 01:56:03.774433 9339 tablet_bootstrap.cc:654] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:03.809228 10130 raft_consensus.cc:493] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.809733 10130 raft_consensus.cc:515] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.814057 10143 raft_consensus.cc:493] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:03.814584 10143 raft_consensus.cc:515] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.820050 10130 leader_election.cc:290] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:03.820760 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:03.808630 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:03.821736 9210 raft_consensus.cc:3060] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:03.824120 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "880d77d42c394e6a9cf0b3768a915402" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:03.823710 9003 leader_election.cc:343] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.824113 10143 leader_election.cc:290] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:03.825659 9003 leader_election.cc:343] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.823837 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 272) took 2198 ms. Trace:
I20260926 01:56:03.844312 10186 rpcz_store.cc:276] 0926 01:56:01.625143 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.625230 (+ 87us) service_pool.cc:224] Handling call
0926 01:56:02.794942 (+1169712us) ts_tablet_manager.cc:1151] Deleting tablet f100eb5dc9a8452e943d4cce5284bc8a
0926 01:56:02.794999 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.242512 (+447513us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.609176 (+366664us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.823741 (+214565us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168853}
I20260926 01:56:03.849418 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.853672 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet f100eb5dc9a8452e943d4cce5284bc8a (table default.Table_37930b35bf904b5aabec0d568896d39b [id=b6747abf32974ff4a3bf5da14e923b1e]) successfully deleted
W20260926 01:56:03.854007 8843 leader_election.cc:343] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.866854 10186 tablet_replica.cc:333] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:03.742290 9711 rpcz_store.cc:276] 0926 01:55:55.366403 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:55.381705 (+ 15302us) service_pool.cc:224] Handling call
0926 01:56:03.041601 (+7659896us) ts_tablet_manager.cc:1151] Deleting tablet 17b8b0d54f5645e5963f1de217f94c84
0926 01:56:03.065251 (+ 23650us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.239403 (+174152us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.620241 (+380838us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:03.741835 (+121594us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7659337}
I20260926 01:56:03.877014 9341 tablet_bootstrap.cc:492] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:03.877092 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 17b8b0d54f5645e5963f1de217f94c84 (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]) successfully deleted
I20260926 01:56:03.877537 9341 ts_tablet_manager.cc:1403] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.390s user 0.002s sys 0.008s
I20260926 01:56:03.880288 9341 raft_consensus.cc:359] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.880970 9341 raft_consensus.cc:385] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.881220 9341 raft_consensus.cc:740] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:03.881853 9341 consensus_queue.cc:260] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.889173 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 188) took 7755 ms. Trace:
I20260926 01:56:03.889354 9340 tablet_bootstrap.cc:492] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:03.889791 9340 ts_tablet_manager.cc:1403] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.586s user 0.007s sys 0.003s
I20260926 01:56:03.894740 9341 ts_tablet_manager.cc:1434] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.017s user 0.005s sys 0.004s
I20260926 01:56:03.895457 10186 raft_consensus.cc:2243] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:03.897341 8534 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 310) took 1161 ms. Trace:
I20260926 01:56:03.897711 8534 rpcz_store.cc:276] 0926 01:56:02.735604 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.482381 (+746777us) service_pool.cc:224] Handling call
0926 01:56:03.526920 (+ 44539us) spinlock_profiling.cc:243] Waited 43.7 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607566 00007f717c1b2034 00007f717c1b1fc2 00007f717c1b1ef1 00007f717c1b1e6c 00007f717c1b1122 00007f717a45c09c 00007f717a45b4d6
0926 01:56:03.642846 (+115926us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:03.645830 (+ 2984us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:03.896604 (+250774us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:03.896955 (+ 351us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:03.897270 (+ 315us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:03.755316 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:03.755851 (+ 535us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:03.755882 (+ 31us) write_op.cc:435] Acquired schema lock
0926 01:56:03.755898 (+ 16us) tablet.cc:662] Decoding operations
0926 01:56:03.756251 (+ 353us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:03.756314 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:03.756341 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:03.756593 (+ 252us) tablet.cc:701] Row locks acquired
0926 01:56:03.756607 (+ 14us) write_op.cc:260] PREPARE: finished
0926 01:56:03.756765 (+ 158us) write_op.cc:270] Start()
0926 01:56:03.756868 (+ 103us) write_op.cc:276] Timestamp: P: 1790387763756699 usec, L: 0
0926 01:56:03.756883 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:56:03.757576 (+ 693us) log.cc:844] Serialized 433 byte log entry
0926 01:56:03.894146 (+136570us) op_driver.cc:464] REPLICATION: finished
0926 01:56:03.894493 (+ 347us) write_op.cc:301] APPLY: starting
0926 01:56:03.894573 (+ 80us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:03.894721 (+ 148us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:03.894853 (+ 132us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:03.895390 (+ 537us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:03.895542 (+ 152us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:03.895586 (+ 44us) write_op.cc:312] APPLY: finished
0926 01:56:03.895822 (+ 236us) log.cc:844] Serialized 29 byte log entry
0926 01:56:03.896008 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:03.896083 (+ 75us) write_op.cc:454] Released schema lock
0926 01:56:03.896126 (+ 43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":100573568,"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":1854,"apply.run_wall_time_us":1855,"num_ops":2,"prepare.queue_time_us":108070,"prepare.run_cpu_time_us":2735,"prepare.run_wall_time_us":143560,"raft.queue_time_us":330,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":305,"replication_time_us":137109,"thread_start_us":107738,"threads_started":1}]]}
I20260926 01:56:03.900532 9340 raft_consensus.cc:359] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.901224 9340 raft_consensus.cc:385] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.901460 9340 raft_consensus.cc:740] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:03.902231 9340 consensus_queue.cc:260] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.904839 9340 ts_tablet_manager.cc:1434] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:56:03.912784 9340 tablet_bootstrap.cc:492] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:03.895982 9341 tablet_bootstrap.cc:492] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:03.916066 10186 raft_consensus.cc:2272] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:03.918959 10186 ts_tablet_manager.cc:1916] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:03.919242 9340 tablet_bootstrap.cc:654] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:03.726075 10262 raft_consensus.cc:515] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:03.931769 9006 leader_election.cc:343] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.932268 9006 leader_election.cc:304] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:03.932935 10127 raft_consensus.cc:2749] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:03.939417 9003 consensus_peers.cc:597] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:03.941233 9340 tablet_bootstrap.cc:492] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:03.941783 9340 ts_tablet_manager.cc:1403] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20260926 01:56:03.944450 9340 raft_consensus.cc:359] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.945281 9340 raft_consensus.cc:385] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.945576 9340 raft_consensus.cc:740] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:03.946269 9340 consensus_queue.cc:260] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.948627 9340 ts_tablet_manager.cc:1434] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 01:56:03.954047 9340 tablet_bootstrap.cc:492] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:03.889545 9711 rpcz_store.cc:276] 0926 01:55:56.133549 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.133662 (+ 113us) service_pool.cc:224] Handling call
0926 01:56:03.888832 (+7755170us) ts_tablet_manager.cc:1151] Deleting tablet 3b6daaf274894989b90da364f5da5250
0926 01:56:03.888885 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:03.889101 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7544772}
W20260926 01:56:03.959635 8527 catalog_manager.cc:5030] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): delete failed for tablet 3b6daaf274894989b90da364f5da5250 because the tablet was not found. No further retry: Not found: Tablet not found: 3b6daaf274894989b90da364f5da5250
I20260926 01:56:03.959915 9711 tablet_replica.cc:333] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:03.960758 9711 raft_consensus.cc:2243] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:03.961313 9711 raft_consensus.cc:2272] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:03.963747 10262 leader_election.cc:290] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:03.964541 9341 tablet_bootstrap.cc:654] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:03.965207 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:03.966055 9214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:03.967073 8843 leader_election.cc:343] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:03.967103 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "880d77d42c394e6a9cf0b3768a915402" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:03.966632 9340 tablet_bootstrap.cc:654] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:03.970983 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:03.972054 9072 raft_consensus.cc:3060] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:03.977959 9210 raft_consensus.cc:2468] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 2.
I20260926 01:56:03.986608 9340 tablet_bootstrap.cc:492] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:03.987154 9340 ts_tablet_manager.cc:1403] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.003s
I20260926 01:56:03.987270 9341 tablet_bootstrap.cc:492] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:03.987754 9341 ts_tablet_manager.cc:1403] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.008s
I20260926 01:56:03.988181 9339 tablet_bootstrap.cc:492] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:03.988639 9339 ts_tablet_manager.cc:1403] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.422s user 0.007s sys 0.002s
I20260926 01:56:03.990542 9341 raft_consensus.cc:359] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.991358 9341 raft_consensus.cc:385] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:03.991676 9341 raft_consensus.cc:740] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:03.992810 9341 consensus_queue.cc:260] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.995498 9341 ts_tablet_manager.cc:1434] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 01:56:03.996459 9341 tablet_bootstrap.cc:492] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:03.999207 9340 raft_consensus.cc:359] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:03.999933 9340 raft_consensus.cc:385] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.000301 9340 raft_consensus.cc:740] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.001037 9340 consensus_queue.cc:260] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.003116 9340 ts_tablet_manager.cc:1434] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260926 01:56:04.004122 9340 tablet_bootstrap.cc:492] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:04.004699 9339 raft_consensus.cc:359] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:04.005092 9006 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:56:04.005375 9339 raft_consensus.cc:385] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.005519 9006 leader_election.cc:304] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:04.005628 9339 raft_consensus.cc:740] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.006860 9341 tablet_bootstrap.cc:654] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.006752 10127 raft_consensus.cc:2749] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:04.014655 9006 leader_election.cc:343] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.015146 9006 leader_election.cc:304] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:04.016247 10127 raft_consensus.cc:2749] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:04.024953 9340 tablet_bootstrap.cc:654] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.044713 9341 tablet_bootstrap.cc:492] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:04.045288 9341 ts_tablet_manager.cc:1403] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.007s
I20260926 01:56:04.047129 9187 tablet_service.cc:1511] Processing CreateTablet for tablet 91964074a37441db8b09f78720159586 (DEFAULT_TABLE table=default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.048331 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91964074a37441db8b09f78720159586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.055420 9341 raft_consensus.cc:359] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.056295 9341 raft_consensus.cc:385] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.056596 9341 raft_consensus.cc:740] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.057410 9341 consensus_queue.cc:260] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.060034 9341 ts_tablet_manager.cc:1434] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260926 01:56:04.061987 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:04.062623 8537 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e
I20260926 01:56:04.063302 9341 tablet_bootstrap.cc:492] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:04.064428 8880 tablet_service.cc:1558] Processing DeleteTablet for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:04.064225 8537 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 297) took 2507 ms. Trace:
I20260926 01:56:04.065124 9190 tablet_service.cc:1558] Processing DeleteTablet for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:42408
W20260926 01:56:04.066190 8524 catalog_manager.cc:5043] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): delete failed for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e already in progress: creating tablet
W20260926 01:56:04.066558 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 consensus state has no leader
I20260926 01:56:04.068369 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:04.069156 10262 raft_consensus.cc:493] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.069664 10262 raft_consensus.cc:515] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.064814 8537 rpcz_store.cc:276] 0926 01:56:01.556412 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.556610 (+ 198us) service_pool.cc:224] Handling call
0926 01:56:01.608369 (+ 51759us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607876 00007f717c1b6b34 00007f717c1b6ac2 00007f717c1b69f1 00007f717c1b696c 00007f717c1b5c22 00007f717a45c09c 00007f717a45b4d6
0926 01:56:01.609895 (+ 1526us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:56:01.610422 (+ 527us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:56:01.610583 (+ 161us) catalog_manager.cc:2863] Locking tablets
0926 01:56:04.039295 (+2428712us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:56:04.062221 (+ 22926us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:56:04.062423 (+ 202us) catalog_manager.cc:2911] Committing in-memory state
0926 01:56:04.062537 (+ 114us) catalog_manager.cc:2915] Aborting table tasks
0926 01:56:04.064154 (+ 1617us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:04.047134 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:04.047606 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:04.047631 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:56:04.047643 (+ 12us) tablet.cc:662] Decoding operations
0926 01:56:04.048001 (+ 358us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:04.048042 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:04.048071 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:04.048380 (+ 309us) tablet.cc:701] Row locks acquired
0926 01:56:04.048398 (+ 18us) write_op.cc:260] PREPARE: finished
0926 01:56:04.048527 (+ 129us) write_op.cc:270] Start()
0926 01:56:04.048624 (+ 97us) write_op.cc:276] Timestamp: P: 1790387764048443 usec, L: 0
0926 01:56:04.048643 (+ 19us) op_driver.cc:348] REPLICATION: starting
0926 01:56:04.049364 (+ 721us) log.cc:844] Serialized 728 byte log entry
0926 01:56:04.058645 (+ 9281us) op_driver.cc:464] REPLICATION: finished
0926 01:56:04.059038 (+ 393us) write_op.cc:301] APPLY: starting
0926 01:56:04.059097 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:04.059224 (+ 127us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:04.059348 (+ 124us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:04.059607 (+ 259us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:04.059750 (+ 143us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:56:04.059798 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:56:04.059986 (+ 188us) log.cc:844] Serialized 29 byte log entry
0926 01:56:04.060135 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:04.060223 (+ 88us) write_op.cc:454] Released schema lock
0926 01:56:04.060257 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":115969920,"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":1421,"apply.run_wall_time_us":1420,"num_ops":2,"prepare.queue_time_us":5968,"raft.queue_time_us":86,"raft.run_cpu_time_us":694,"raft.run_wall_time_us":694,"replication_time_us":9900}]]}
I20260926 01:56:04.072487 10262 leader_election.cc:290] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.073056 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f49a5a10ea4f7a821d89f18e3af560" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.074270 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f49a5a10ea4f7a821d89f18e3af560" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.075587 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
W20260926 01:56:04.075894 8843 leader_election.cc:343] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:04.077133 8843 leader_election.cc:343] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.077512 8843 leader_election.cc:304] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:04.078162 10262 raft_consensus.cc:2749] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:04.078649 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 3 got 0
I20260926 01:56:04.080117 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet f2ee16b0c3e84936abba8248f194786a (table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c]). Expected version 1 got 0
I20260926 01:56:04.081272 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]). Expected version 1 got 0
I20260926 01:56:04.082062 8538 catalog_manager.cc:5838] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } }
I20260926 01:56:04.083678 9711 ts_tablet_manager.cc:1916] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:04.088935 10192 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.089470 10192 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.091641 10192 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:04.094669 9155 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:04.099711 9072 raft_consensus.cc:2468] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:04.101053 9340 tablet_bootstrap.cc:492] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:04.095436 9341 tablet_bootstrap.cc:654] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.101281 8843 leader_election.cc:304] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:04.101585 9340 ts_tablet_manager.cc:1403] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.098s user 0.021s sys 0.016s
I20260926 01:56:04.102061 10262 raft_consensus.cc:2804] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:04.006256 9339 consensus_queue.cc:260] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.112533 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:04.113492 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]). Expected version 1 got 0
I20260926 01:56:04.115561 10262 raft_consensus.cc:697] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:04.116600 9341 tablet_bootstrap.cc:492] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:04.117058 9341 ts_tablet_manager.cc:1403] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20260926 01:56:04.116611 10262 consensus_queue.cc:237] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.120566 9340 raft_consensus.cc:359] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.122339 9340 raft_consensus.cc:385] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.122709 9340 raft_consensus.cc:740] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.130880 9339 ts_tablet_manager.cc:1434] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.142s user 0.007s sys 0.001s
I20260926 01:56:04.130511 9341 raft_consensus.cc:359] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.131234 9341 raft_consensus.cc:385] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.131492 9341 raft_consensus.cc:740] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.132007 9339 tablet_bootstrap.cc:492] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:04.131574 9340 consensus_queue.cc:260] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.138310 10174 raft_consensus.cc:493] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.138824 10174 raft_consensus.cc:515] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.139817 8538 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:04.140928 10174 leader_election.cc:290] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.142028 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 91964074a37441db8b09f78720159586 (DEFAULT_TABLE table=default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.142604 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4623456312284a698ae69d4054909647" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.143289 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91964074a37441db8b09f78720159586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.143275 9210 raft_consensus.cc:2468] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:04.143187 9341 consensus_queue.cc:260] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.143463 9048 tablet_service.cc:1511] Processing CreateTablet for tablet 91964074a37441db8b09f78720159586 (DEFAULT_TABLE table=default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.144706 9048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91964074a37441db8b09f78720159586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.144847 9190 tablet_service.cc:1511] Processing CreateTablet for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c (DEFAULT_TABLE table=default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.145963 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.146075 9341 ts_tablet_manager.cc:1434] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260926 01:56:04.147059 9341 tablet_bootstrap.cc:492] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:04.151456 8843 leader_election.cc:304] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:04.152287 10174 raft_consensus.cc:2804] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:04.152743 10174 raft_consensus.cc:493] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:04.153129 10174 raft_consensus.cc:3060] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.157789 9341 tablet_bootstrap.cc:654] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.164776 9340 ts_tablet_manager.cc:1434] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.063s user 0.007s sys 0.001s
I20260926 01:56:04.167467 9340 tablet_bootstrap.cc:492] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:04.169734 9006 leader_election.cc:343] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.170166 9006 leader_election.cc:304] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:04.173004 10130 raft_consensus.cc:2749] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:04.168560 10174 raft_consensus.cc:515] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.179067 10174 leader_election.cc:290] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.179383 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4623456312284a698ae69d4054909647" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.180020 9072 raft_consensus.cc:2468] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:04.188920 9341 tablet_bootstrap.cc:492] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:04.194911 9340 tablet_bootstrap.cc:654] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.214221 9340 tablet_bootstrap.cc:492] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:04.214802 9340 ts_tablet_manager.cc:1403] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.003s
I20260926 01:56:04.217546 9340 raft_consensus.cc:359] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.218400 9340 raft_consensus.cc:385] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.218734 9340 raft_consensus.cc:740] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.219506 9340 consensus_queue.cc:260] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.222180 9340 ts_tablet_manager.cc:1434] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:56:04.225492 9340 tablet_bootstrap.cc:492] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:04.225781 9265 heartbeater.cc:499] Master 127.8.76.188:38143 was elected leader, sending a full tablet report...
I20260926 01:56:04.244144 9340 tablet_bootstrap.cc:654] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.261343 9340 tablet_bootstrap.cc:492] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:04.261893 9340 ts_tablet_manager.cc:1403] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20260926 01:56:04.264744 9340 raft_consensus.cc:359] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.265470 9340 raft_consensus.cc:385] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.265713 9340 raft_consensus.cc:740] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:56:04.266654 9152 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 43714.9us
I20260926 01:56:04.266346 9340 consensus_queue.cc:260] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.271875 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4623456312284a698ae69d4054909647" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:04.272473 9072 raft_consensus.cc:3060] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.272696 9340 ts_tablet_manager.cc:1434] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260926 01:56:04.273695 9340 tablet_bootstrap.cc:492] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:04.275837 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17b8b0d54f5645e5963f1de217f94c84" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:04.280332 8880 tablet_service.cc:1511] Processing CreateTablet for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c (DEFAULT_TABLE table=default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.280723 9217 raft_consensus.cc:3060] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.281474 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.286497 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4623456312284a698ae69d4054909647" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:04.287058 9210 raft_consensus.cc:3060] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.287863 9072 raft_consensus.cc:2468] T 4623456312284a698ae69d4054909647 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:04.289196 8843 leader_election.cc:304] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:04.290257 10174 raft_consensus.cc:2804] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:04.298774 9210 raft_consensus.cc:2468] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:04.298727 10287 consensus_queue.cc:1048] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:04.313055 10174 raft_consensus.cc:697] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:04.314146 10174 consensus_queue.cc:237] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.318776 9340 tablet_bootstrap.cc:654] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.323421 10178 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:04.324505 9990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45746 (request call id 0) took 5951 ms. Trace:
I20260926 01:56:04.324250 10178 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:04.335361 10174 raft_consensus.cc:493] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.336017 10174 raft_consensus.cc:515] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.337842 10140 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 5 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:04.338702 10140 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:04.339705 10248 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:04.340499 10248 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:04.340595 10174 leader_election.cc:290] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.342108 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18ddaa6f0e748cf930e29cbca3844b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.342307 9340 tablet_bootstrap.cc:492] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:04.342916 9340 ts_tablet_manager.cc:1403] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.003s
I20260926 01:56:04.343338 9072 raft_consensus.cc:2468] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:04.344620 8843 leader_election.cc:304] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:04.345486 10174 raft_consensus.cc:2804] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:04.345880 10174 raft_consensus.cc:493] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:04.346243 10174 raft_consensus.cc:3060] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.353078 9340 raft_consensus.cc:359] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.353967 9340 raft_consensus.cc:385] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.358611 9340 raft_consensus.cc:740] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.359561 9340 consensus_queue.cc:260] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.365765 9340 ts_tablet_manager.cc:1434] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20260926 01:56:04.357868 10174 raft_consensus.cc:515] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.368017 10174 leader_election.cc:290] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.369275 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18ddaa6f0e748cf930e29cbca3844b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:04.369890 9211 raft_consensus.cc:3060] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.379293 10178 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:04.384807 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18ddaa6f0e748cf930e29cbca3844b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.407883 9341 ts_tablet_manager.cc:1403] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.261s user 0.015s sys 0.001s
I20260926 01:56:04.410683 9341 raft_consensus.cc:359] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.411408 9341 raft_consensus.cc:385] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.411768 9341 raft_consensus.cc:740] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.412560 9341 consensus_queue.cc:260] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.414634 9341 ts_tablet_manager.cc:1434] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 01:56:04.415673 9341 tablet_bootstrap.cc:492] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:04.416241 9152 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:04.378186 9211 raft_consensus.cc:2468] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:04.419064 8843 leader_election.cc:304] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:04.419922 10174 raft_consensus.cc:2804] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:04.421425 10174 raft_consensus.cc:697] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:04.422403 10174 consensus_queue.cc:237] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.433897 9341 tablet_bootstrap.cc:654] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.440289 9187 tablet_service.cc:1511] Processing CreateTablet for tablet 70d2447335b34731a4cb2cd0f3b4bf13 (DEFAULT_TABLE table=default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.441645 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d2447335b34731a4cb2cd0f3b4bf13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.440392 9192 tablet_service.cc:1511] Processing CreateTablet for tablet ea75f309937e4ee49b35558117847b05 (DEFAULT_TABLE table=default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.444021 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea75f309937e4ee49b35558117847b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.460352 9341 tablet_bootstrap.cc:492] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:04.463985 10296 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 01:56:04.471807 10192 raft_consensus.cc:2749] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:04.474658 9341 ts_tablet_manager.cc:1403] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.002s
I20260926 01:56:04.485297 9341 raft_consensus.cc:359] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.486047 9341 raft_consensus.cc:385] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.486325 9341 raft_consensus.cc:740] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.486987 9341 consensus_queue.cc:260] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.489032 9341 ts_tablet_manager.cc:1434] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.014s user 0.003s sys 0.001s
I20260926 01:56:04.489962 9341 tablet_bootstrap.cc:492] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:04.505890 8535 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 312) took 1535 ms. Trace:
I20260926 01:56:04.506256 8535 rpcz_store.cc:276] 0926 01:56:02.970473 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.491282 (+520809us) service_pool.cc:224] Handling call
0926 01:56:03.526208 (+ 34926us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60a0e1 00007f717c1d2d37 00007f717c1d2cc2 00007f717c1d2bf1 00007f717c1d2b6c 00007f717c1d1e22 00007f717a45c09c 00007f717a45b4d6
0926 01:56:04.505807 (+979599us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77761920}
I20260926 01:56:04.507208 9341 tablet_bootstrap.cc:654] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.526345 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f]). Expected version 3 got 0
I20260926 01:56:04.527314 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]). Expected version 1 got 0
I20260926 01:56:04.528163 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:04.528882 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet f2ee16b0c3e84936abba8248f194786a (table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c]). Expected version 1 got 0
I20260926 01:56:04.529580 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:04.530609 8531 catalog_manager.cc:5838] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } }
I20260926 01:56:04.540180 8879 tablet_service.cc:1511] Processing CreateTablet for tablet 70d2447335b34731a4cb2cd0f3b4bf13 (DEFAULT_TABLE table=default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.541743 8882 tablet_service.cc:1511] Processing CreateTablet for tablet ea75f309937e4ee49b35558117847b05 (DEFAULT_TABLE table=default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.544188 10293 raft_consensus.cc:493] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.544703 10293 raft_consensus.cc:515] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.548023 10293 leader_election.cc:290] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.558362 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d56f3a5dfed4ac78f1da92e22db558a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.560672 9211 raft_consensus.cc:2468] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:04.561923 8843 leader_election.cc:304] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:04.572619 10262 raft_consensus.cc:2804] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:04.573199 10262 raft_consensus.cc:493] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:04.573506 10262 raft_consensus.cc:3060] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.581830 10262 raft_consensus.cc:515] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.588008 9962 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "49bd320bf73f4252804ec867ec077510" instance_seqno: 1790387734231111) as {username='slave'} at 127.8.76.129:59597; Asking this server to re-register.
I20260926 01:56:04.589531 8625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 6 candidate_status { last_received { term: 5 index: 283 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:04.591193 10293 raft_consensus.cc:493] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.592257 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea75f309937e4ee49b35558117847b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.593431 9340 tablet_bootstrap.cc:492] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:04.635001 10293 raft_consensus.cc:515] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.636047 9340 tablet_bootstrap.cc:654] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.637889 8879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d2447335b34731a4cb2cd0f3b4bf13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.652411 9340 tablet_bootstrap.cc:492] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:04.652920 9340 ts_tablet_manager.cc:1403] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.000s
I20260926 01:56:04.656136 9340 raft_consensus.cc:359] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.656883 9340 raft_consensus.cc:385] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.657150 9340 raft_consensus.cc:740] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.664654 9340 consensus_queue.cc:260] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.666162 10174 raft_consensus.cc:493] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.666656 10174 raft_consensus.cc:515] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.667163 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d56f3a5dfed4ac78f1da92e22db558a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:04.667738 9211 raft_consensus.cc:3060] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.668916 10293 leader_election.cc:290] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.669732 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "880d77d42c394e6a9cf0b3768a915402" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.671154 10174 leader_election.cc:290] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:04.671698 8843 leader_election.cc:343] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.673341 9340 ts_tablet_manager.cc:1434] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s
I20260926 01:56:04.673218 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.674319 9340 tablet_bootstrap.cc:492] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:04.674695 8843 leader_election.cc:343] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.675961 9211 raft_consensus.cc:2468] T 6d56f3a5dfed4ac78f1da92e22db558a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:04.677098 8843 leader_election.cc:304] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:04.677872 10174 raft_consensus.cc:2804] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:04.678491 10287 raft_consensus.cc:493] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 01:56:04.682946 10241 log.cc:927] Time spent T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Append to log took a long time: real 0.233s user 0.001s sys 0.001s
I20260926 01:56:04.678937 10287 raft_consensus.cc:515] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.690369 10287 leader_election.cc:290] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.679526 10174 raft_consensus.cc:697] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:04.691915 10174 consensus_queue.cc:237] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.697569 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c8761a6c3f7435193a91bfc4fe7fa86" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.699034 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18ddaa6f0e748cf930e29cbca3844b1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:56:04.699425 8843 leader_election.cc:343] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.699620 9072 raft_consensus.cc:3060] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.702283 9339 tablet_bootstrap.cc:700] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2: Time spent opening tablet: real 0.439s user 0.001s sys 0.000s
I20260926 01:56:04.702849 9046 tablet_service.cc:1511] Processing CreateTablet for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c (DEFAULT_TABLE table=default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.703920 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.707049 9340 tablet_bootstrap.cc:654] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.715272 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "880d77d42c394e6a9cf0b3768a915402" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.715945 9070 raft_consensus.cc:2468] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:04.729967 9072 raft_consensus.cc:2468] T e18ddaa6f0e748cf930e29cbca3844b1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:04.735903 9339 tablet_bootstrap.cc:654] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.628500 9959 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bc00feec3f564608851f0112e97ab532" instance_seqno: 1790387739152191) as {username='slave'} at 127.8.76.131:60645; Asking this server to re-register.
I20260926 01:56:04.747982 9265 heartbeater.cc:461] Registering TS with master...
I20260926 01:56:04.754910 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d56f3a5dfed4ac78f1da92e22db558a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:04.755714 9072 raft_consensus.cc:3060] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.699750 10219 raft_consensus.cc:493] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.667074 10262 leader_election.cc:290] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.774890 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.775560 9070 raft_consensus.cc:2468] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:04.779486 9959 ts_manager.cc:194] Registered new tserver with Master: bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:04.781754 8843 consensus_peers.cc:597] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:04.782449 8843 leader_election.cc:304] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:04.772521 10219 raft_consensus.cc:515] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.783967 8843 leader_election.cc:304] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:04.785151 10219 leader_election.cc:290] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.785597 10287 raft_consensus.cc:2804] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:04.785969 10287 raft_consensus.cc:493] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:04.786280 10287 raft_consensus.cc:3060] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.787737 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c8761a6c3f7435193a91bfc4fe7fa86" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.790644 10293 raft_consensus.cc:2804] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:04.791028 10293 raft_consensus.cc:493] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:04.791347 10293 raft_consensus.cc:3060] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.787632 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d56f3a5dfed4ac78f1da92e22db558a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.794165 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d068e93fd374a6091d7473db069a7fa" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:04.796119 8906 raft_consensus.cc:1275] T 681900e0fac74cdabaea1d8a7fb4a189 P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Refusing update from remote peer d20f26c9295641ecbf77d290ca206bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260926 01:56:04.797597 9003 leader_election.cc:343] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:04.796150 8843 leader_election.cc:343] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.798923 8843 leader_election.cc:304] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:04.800221 10192 raft_consensus.cc:493] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:04.800737 10192 raft_consensus.cc:515] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:04.801405 9048 consensus_meta.cc:287] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2: Time spent flushing consensus metadata: real 0.612s user 0.009s sys 0.002s
I20260926 01:56:04.797708 10219 consensus_queue.cc:1048] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260926 01:56:04.804279 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:04.804634 10192 leader_election.cc:290] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:04.805146 10127 consensus_queue.cc:1048] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:04.808827 10293 raft_consensus.cc:515] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.817059 10174 raft_consensus.cc:2749] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:04.813998 10287 raft_consensus.cc:515] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.822124 10287 leader_election.cc:290] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.820235 10293 leader_election.cc:290] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.828071 8952 heartbeater.cc:461] Registering TS with master...
I20260926 01:56:04.835099 9341 tablet_bootstrap.cc:492] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:04.835594 9341 ts_tablet_manager.cc:1403] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.346s user 0.052s sys 0.081s
I20260926 01:56:04.838353 9341 raft_consensus.cc:359] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.839484 9341 raft_consensus.cc:385] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.839759 9341 raft_consensus.cc:740] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:04.840493 9341 consensus_queue.cc:260] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.842300 9341 ts_tablet_manager.cc:1434] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260926 01:56:04.843191 9341 tablet_bootstrap.cc:492] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:04.847218 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:04.847790 9068 raft_consensus.cc:3060] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.834331 10186 ts_tablet_manager.cc:1929] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:04.849241 10186 log.cc:1199] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/cbf8818e65d74a32ba21dcb8a9a56a62
I20260926 01:56:04.850173 10186 ts_tablet_manager.cc:1950] T cbf8818e65d74a32ba21dcb8a9a56a62 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:04.865734 9072 raft_consensus.cc:2468] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:04.848910 9341 tablet_bootstrap.cc:654] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:04.870811 9910 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.528s (SENT)
I20260926 01:56:04.821754 9962 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" instance_seqno: 1790387736721407) as {username='slave'} at 127.8.76.130:42977; Asking this server to re-register.
W20260926 01:56:04.920331 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 50024.6us
W20260926 01:56:04.932394 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 115972us
I20260926 01:56:04.924183 9210 raft_consensus.cc:1275] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Refusing update from remote peer d20f26c9295641ecbf77d290ca206bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260926 01:56:04.934216 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:04.893791 9341 tablet_bootstrap.cc:492] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:04.937808 9341 ts_tablet_manager.cc:1403] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.006s
I20260926 01:56:04.937384 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "880d77d42c394e6a9cf0b3768a915402" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:04.940431 9341 raft_consensus.cc:359] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.941035 9341 raft_consensus.cc:385] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:04.941293 9341 raft_consensus.cc:740] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:56:04.942502 8843 leader_election.cc:343] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:04.943121 8843 leader_election.cc:343] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:04.941900 9341 consensus_queue.cc:260] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.945541 9341 ts_tablet_manager.cc:1434] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:56:04.946763 9341 tablet_bootstrap.cc:492] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:04.852870 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 277) took 3200 ms. Trace:
I20260926 01:56:04.534962 8535 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_e04a4278262e4d7f8d7a94ddf8666bbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:04.986552 8535 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e04a4278262e4d7f8d7a94ddf8666bbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:04.543254 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 6 candidate_status { last_received { term: 5 index: 283 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:04.991392 8550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 0) took 448 ms (client timeout 528 ms). Trace:
W20260926 01:56:04.991840 8550 rpcz_store.cc:269] 0926 01:56:04.542836 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:04.542986 (+ 150us) service_pool.cc:224] Handling call
0926 01:56:04.991159 (+448173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:04.993734 8880 tablet_service.cc:1511] Processing CreateTablet for tablet d7e2b22ca59745879270e8dc20f2a4fa (DEFAULT_TABLE table=default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:04.995357 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:04.998500 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]). Expected version 1 got 0
I20260926 01:56:05.000021 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e2b22ca59745879270e8dc20f2a4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.978396 8549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 6 candidate_status { last_received { term: 5 index: 283 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:05.020963 10174 raft_consensus.cc:493] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:05.021552 10174 raft_consensus.cc:515] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.021735 8531 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:05.023579 10174 leader_election.cc:290] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:05.047612 8874 tablet_service.cc:1511] Processing CreateTablet for tablet 2a093fc10b604aa491b6e0f9596a589d (DEFAULT_TABLE table=default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.048936 8874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a093fc10b604aa491b6e0f9596a589d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.051540 8882 tablet_service.cc:1511] Processing CreateTablet for tablet b25107dd5c104c83a2a21847bb2fc24f (DEFAULT_TABLE table=default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.052642 8882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25107dd5c104c83a2a21847bb2fc24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.088620 8884 tablet_service.cc:1511] Processing CreateTablet for tablet 08b1d98089fd499a8e1ee217b561a801 (DEFAULT_TABLE table=default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.089728 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b1d98089fd499a8e1ee217b561a801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.952948 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d068e93fd374a6091d7473db069a7fa" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:04.971392 10191 raft_consensus.cc:493] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:05.095044 10191 raft_consensus.cc:515] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.097227 10191 leader_election.cc:290] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:04.983011 9187 tablet_service.cc:1511] Processing CreateTablet for tablet b25107dd5c104c83a2a21847bb2fc24f (DEFAULT_TABLE table=default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.099100 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25107dd5c104c83a2a21847bb2fc24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.098968 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f49a5a10ea4f7a821d89f18e3af560" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:05.099637 8906 raft_consensus.cc:2468] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:56:05.103158 9152 leader_election.cc:304] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:05.105044 10326 raft_consensus.cc:2804] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:05.105407 10326 raft_consensus.cc:493] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:05.105712 10326 raft_consensus.cc:3060] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:04.997476 9217 raft_consensus.cc:3060] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:05.115273 10318 consensus_queue.cc:1048] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:05.011700 9190 tablet_service.cc:1511] Processing CreateTablet for tablet d7e2b22ca59745879270e8dc20f2a4fa (DEFAULT_TABLE table=default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.029443 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fdb7a7328c54031bd3afac1636d0f75" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:05.119402 9216 raft_consensus.cc:2468] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:05.119323 9184 tablet_service.cc:1511] Processing CreateTablet for tablet 8aae2a23d3fe478e928757db4b30f5ca (DEFAULT_TABLE table=default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.045936 9192 tablet_service.cc:1511] Processing CreateTablet for tablet 2a093fc10b604aa491b6e0f9596a589d (DEFAULT_TABLE table=default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.085153 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 08b1d98089fd499a8e1ee217b561a801 (DEFAULT_TABLE table=default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.120566 9184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aae2a23d3fe478e928757db4b30f5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.121259 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b1d98089fd499a8e1ee217b561a801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.127465 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e2b22ca59745879270e8dc20f2a4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.128343 8843 leader_election.cc:304] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:05.129048 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a093fc10b604aa491b6e0f9596a589d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.129249 10318 raft_consensus.cc:2804] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:05.129596 10318 raft_consensus.cc:493] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:05.129899 10318 raft_consensus.cc:3060] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:05.139796 10191 raft_consensus.cc:493] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:05.140329 10191 raft_consensus.cc:515] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.143987 10191 leader_election.cc:290] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:05.148669 10174 consensus_queue.cc:1048] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:05.166968 8881 tablet_service.cc:1511] Processing CreateTablet for tablet 8aae2a23d3fe478e928757db4b30f5ca (DEFAULT_TABLE table=default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.168196 8881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aae2a23d3fe478e928757db4b30f5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.178444 10326 raft_consensus.cc:515] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:04.866634 10311 raft_consensus.cc:493] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc00feec3f564608851f0112e97ab532)
I20260926 01:56:05.191859 10311 raft_consensus.cc:515] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.194137 10311 leader_election.cc:290] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:04.887533 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92896db04df346bf91f732b1ce89cd23" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:04.896701 10127 consensus_queue.cc:1048] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:04.898679 9048 tablet_service.cc:1511] Processing CreateTablet for tablet 70d2447335b34731a4cb2cd0f3b4bf13 (DEFAULT_TABLE table=default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.196413 9048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d2447335b34731a4cb2cd0f3b4bf13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:04.939592 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "880d77d42c394e6a9cf0b3768a915402" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:05.201663 9070 raft_consensus.cc:3060] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:05.004999 9118 heartbeater.cc:461] Registering TS with master...
I20260926 01:56:05.010171 9711 ts_tablet_manager.cc:1929] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:05.207306 9711 log.cc:1199] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/16f49a5a10ea4f7a821d89f18e3af560
I20260926 01:56:05.207998 9711 ts_tablet_manager.cc:1950] T 16f49a5a10ea4f7a821d89f18e3af560 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:05.013976 9339 tablet_bootstrap.cc:492] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:05.059510 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fdb7a7328c54031bd3afac1636d0f75" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:05.208781 9339 ts_tablet_manager.cc:1403] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 1.077s user 0.015s sys 0.009s
I20260926 01:56:05.208837 9067 raft_consensus.cc:2468] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:04.869275 9068 raft_consensus.cc:2468] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:05.211318 9339 raft_consensus.cc:359] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.211990 9339 raft_consensus.cc:385] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:05.212260 9339 raft_consensus.cc:740] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:05.212977 9339 consensus_queue.cc:260] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.215098 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 195) took 9076 ms. Trace:
I20260926 01:56:05.215411 9711 rpcz_store.cc:276] 0926 01:55:56.138699 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.138874 (+ 175us) service_pool.cc:224] Handling call
0926 01:56:03.959732 (+7820858us) ts_tablet_manager.cc:1151] Deleting tablet 16f49a5a10ea4f7a821d89f18e3af560
0926 01:56:03.959784 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:04.596795 (+637011us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:05.010019 (+413224us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:05.215007 (+204988us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7615084}
I20260926 01:56:05.216617 9339 ts_tablet_manager.cc:1434] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 01:56:05.217018 9711 tablet_replica.cc:333] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:05.218133 9339 tablet_bootstrap.cc:492] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:05.237370 9711 raft_consensus.cc:2243] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:05.237977 9711 raft_consensus.cc:2272] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:05.240242 9711 ts_tablet_manager.cc:1916] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:05.237921 9070 raft_consensus.cc:2468] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:05.243950 9339 tablet_bootstrap.cc:654] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:05.245507 10186 rpcz_store.cc:276] 0926 01:56:01.652864 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.652960 (+ 96us) service_pool.cc:224] Handling call
0926 01:56:03.866636 (+2213676us) ts_tablet_manager.cc:1151] Deleting tablet cbf8818e65d74a32ba21dcb8a9a56a62
0926 01:56:03.866693 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:04.441359 (+574666us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:04.834149 (+392790us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:04.852797 (+ 18648us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2179539}
I20260926 01:56:05.247469 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet cbf8818e65d74a32ba21dcb8a9a56a62 (table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]) successfully deleted
I20260926 01:56:05.280341 9155 leader_election.cc:304] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:05.281999 10191 raft_consensus.cc:2749] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:05.313974 9962 ts_manager.cc:194] Registered new tserver with Master: 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:05.321460 10186 tablet_replica.cc:333] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:05.322566 10186 raft_consensus.cc:2243] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:05.323176 10186 raft_consensus.cc:2272] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:05.325546 10186 ts_tablet_manager.cc:1916] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:05.360081 10327 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:05.363693 10327 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:05.344759 10311 raft_consensus.cc:493] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:05.365888 10311 raft_consensus.cc:515] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.324867 9339 tablet_bootstrap.cc:492] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:05.368724 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f10cb68586b4a73abb86641b90c12f7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:05.369176 9339 ts_tablet_manager.cc:1403] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.151s user 0.012s sys 0.000s
I20260926 01:56:05.369572 9070 raft_consensus.cc:2468] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
W20260926 01:56:05.418501 8879 consensus_meta.cc:287] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: Time spent flushing consensus metadata: real 0.563s user 0.006s sys 0.000s
I20260926 01:56:05.429396 9959 ts_manager.cc:194] Registered new tserver with Master: d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:05.451584 9155 leader_election.cc:343] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 16f49a5a10ea4f7a821d89f18e3af560
I20260926 01:56:05.453150 9155 leader_election.cc:304] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:05.453985 10191 raft_consensus.cc:2804] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:05.454332 10191 raft_consensus.cc:493] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:05.454687 10191 raft_consensus.cc:3060] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:05.453312 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 50529.3us
I20260926 01:56:05.456725 8843 leader_election.cc:304] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:05.418803 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:05.481534 9341 tablet_bootstrap.cc:654] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:05.486346 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 16f49a5a10ea4f7a821d89f18e3af560 (table default.Table_f820adee11bf4557b6119642992da842 [id=d16775efe3d1459a9e1d3152f2b945f5]) successfully deleted
I20260926 01:56:05.487164 10343 raft_consensus.cc:2804] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:05.487752 10343 raft_consensus.cc:697] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:05.496718 8535 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 313) took 1828 ms. Trace:
I20260926 01:56:05.497115 8535 rpcz_store.cc:276] 0926 01:56:03.668186 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:04.507367 (+839181us) service_pool.cc:224] Handling call
0926 01:56:04.534840 (+ 27473us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607566 00007f717c1b2034 00007f717c1b1fc2 00007f717c1b1ef1 00007f717c1b1e6c 00007f717c1b1122 00007f717a45c09c 00007f717a45b4d6
0926 01:56:04.987168 (+452328us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:04.989295 (+ 2127us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:05.495868 (+506573us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:05.496275 (+ 407us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:05.496620 (+ 345us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:05.036992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:05.037715 (+ 723us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:05.037757 (+ 42us) write_op.cc:435] Acquired schema lock
0926 01:56:05.037776 (+ 19us) tablet.cc:662] Decoding operations
0926 01:56:05.038168 (+ 392us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:05.038242 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:05.038267 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:05.039343 (+ 1076us) tablet.cc:701] Row locks acquired
0926 01:56:05.039371 (+ 28us) write_op.cc:260] PREPARE: finished
0926 01:56:05.039516 (+ 145us) write_op.cc:270] Start()
0926 01:56:05.039659 (+ 143us) write_op.cc:276] Timestamp: P: 1790387765039387 usec, L: 0
0926 01:56:05.039686 (+ 27us) op_driver.cc:348] REPLICATION: starting
0926 01:56:05.040369 (+ 683us) log.cc:844] Serialized 433 byte log entry
0926 01:56:05.415936 (+375567us) op_driver.cc:464] REPLICATION: finished
0926 01:56:05.493884 (+ 77948us) write_op.cc:301] APPLY: starting
0926 01:56:05.493969 (+ 85us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:05.494118 (+ 149us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:05.494249 (+ 131us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:05.494809 (+ 560us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:05.494998 (+ 189us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:05.495046 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:56:05.495305 (+ 259us) log.cc:844] Serialized 29 byte log entry
0926 01:56:05.495489 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:05.495566 (+ 77us) write_op.cc:454] Released schema lock
0926 01:56:05.495621 (+ 55us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":60638080,"child_traces":[["op",{"apply.queue_time_us":77675,"num_ops":2,"prepare.queue_time_us":12013,"prepare.run_cpu_time_us":2907,"prepare.run_wall_time_us":3792,"raft.queue_time_us":1581,"raft.run_cpu_time_us":789,"raft.run_wall_time_us":789,"replication_time_us":376086,"thread_start_us":77882,"threads_started":1}]]}
W20260926 01:56:05.502732 8535 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.8.76.131:48005 (request call id 37) took 950 ms (client timeout 1000 ms). Trace:
W20260926 01:56:05.503180 8535 rpcz_store.cc:269] 0926 01:56:04.552636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:05.502473 (+949837us) service_pool.cc:224] Handling call
0926 01:56:05.502652 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:05.505291 8535 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_a82e6ba61ea64aaa8d10956665642633" } new_table_name: "default.Table_85a98e90b8d2451d9b35f8bf4013c05b" modify_external_catalogs: true
I20260926 01:56:05.510389 9267 heartbeater.cc:344] Connected to a master server at 127.8.76.190:44589
I20260926 01:56:05.516623 10326 leader_election.cc:290] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:05.521209 9339 raft_consensus.cc:359] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.521947 9339 raft_consensus.cc:385] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:05.522187 9339 raft_consensus.cc:740] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:05.523051 9339 consensus_queue.cc:260] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.526757 10311 leader_election.cc:290] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:05.509524 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f10cb68586b4a73abb86641b90c12f7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:05.478228 9340 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.036s sys 0.077s
I20260926 01:56:05.520422 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f49a5a10ea4f7a821d89f18e3af560" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:56:05.530755 9340 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.821s user 0.037s sys 0.077s
I20260926 01:56:05.530906 8904 raft_consensus.cc:3060] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:05.533339 9340 tablet_bootstrap.cc:492] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:05.533823 9340 ts_tablet_manager.cc:1403] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.860s user 0.045s sys 0.077s
W20260926 01:56:05.524902 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 68231.1us
I20260926 01:56:05.534991 8843 leader_election.cc:304] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:05.541728 10344 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260926 01:56:05.545702 10346 raft_consensus.cc:2804] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:05.546196 10346 raft_consensus.cc:697] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:05.547060 10346 consensus_queue.cc:237] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.549281 8904 raft_consensus.cc:2468] T 16f49a5a10ea4f7a821d89f18e3af560 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:56:05.551057 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 138930us
W20260926 01:56:05.552055 10308 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.049s sys 0.111s
W20260926 01:56:05.552378 10308 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s user 0.049s sys 0.111s
I20260926 01:56:05.553288 10343 consensus_queue.cc:237] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.554616 10318 raft_consensus.cc:515] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:05.555209 9155 leader_election.cc:343] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 16f49a5a10ea4f7a821d89f18e3af560
I20260926 01:56:05.556643 10318 leader_election.cc:290] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:05.560094 10296 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:56:05.560384 10296 catalog_manager.cc:1601] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: loading cluster ID for follower catalog manager: success
W20260926 01:56:05.564757 9003 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 3571967749844fd7859cf179a93cf403
I20260926 01:56:05.572249 9339 ts_tablet_manager.cc:1434] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.203s user 0.009s sys 0.001s
I20260926 01:56:05.575340 9339 tablet_bootstrap.cc:492] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:05.580263 10327 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:05.581825 9046 tablet_service.cc:1511] Processing CreateTablet for tablet ea75f309937e4ee49b35558117847b05 (DEFAULT_TABLE table=default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.582934 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea75f309937e4ee49b35558117847b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.584673 8549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 350 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:05.586776 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 350 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:05.587419 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef98e01cb9a44408ebf207163f208b5 in term 5.
I20260926 01:56:05.592550 9339 tablet_bootstrap.cc:654] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:05.592816 10336 log.cc:927] Time spent T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2: Append to log took a long time: real 0.070s user 0.001s sys 0.001s
I20260926 01:56:05.598984 9043 tablet_service.cc:1511] Processing CreateTablet for tablet d7e2b22ca59745879270e8dc20f2a4fa (DEFAULT_TABLE table=default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.600247 9043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e2b22ca59745879270e8dc20f2a4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.609174 9339 tablet_bootstrap.cc:492] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:05.609704 9339 ts_tablet_manager.cc:1403] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.004s
I20260926 01:56:05.611380 9051 tablet_service.cc:1511] Processing CreateTablet for tablet b25107dd5c104c83a2a21847bb2fc24f (DEFAULT_TABLE table=default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.611833 9045 tablet_service.cc:1511] Processing CreateTablet for tablet 08b1d98089fd499a8e1ee217b561a801 (DEFAULT_TABLE table=default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.612537 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25107dd5c104c83a2a21847bb2fc24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.613193 9045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b1d98089fd499a8e1ee217b561a801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.614074 9050 tablet_service.cc:1511] Processing CreateTablet for tablet 2a093fc10b604aa491b6e0f9596a589d (DEFAULT_TABLE table=default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.615156 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a093fc10b604aa491b6e0f9596a589d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.588897 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters: ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:05.625859 10327 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260926 01:56:05.633564 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 8aae2a23d3fe478e928757db4b30f5ca (DEFAULT_TABLE table=default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:05.634938 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aae2a23d3fe478e928757db4b30f5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:05.634682 10327 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:05.635233 10327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 5 FOLLOWER]: Advancing to term 6
I20260926 01:56:05.637544 10314 consensus_queue.cc:1048] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:05.639035 10356 tablet.cc:1751] T 8a56044249bb4aaa81903d2a8aeb008e P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260926 01:56:05.643720 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 65730.6us
I20260926 01:56:05.661886 10353 tablet.cc:1751] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260926 01:56:05.655838 9006 leader_election.cc:343] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:05.664367 9006 leader_election.cc:304] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:05.666316 9711 ts_tablet_manager.cc:1929] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:05.666738 9711 log.cc:1199] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/9bac55da5b164c0f981914d39c87c33e
I20260926 01:56:05.667402 9711 ts_tablet_manager.cc:1950] T 9bac55da5b164c0f981914d39c87c33e P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:05.668309 10127 raft_consensus.cc:2749] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:05.672485 9006 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 3571967749844fd7859cf179a93cf403
W20260926 01:56:05.675392 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 119614us
W20260926 01:56:05.682137 9152 leader_election.cc:343] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:05.683532 9152 leader_election.cc:304] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:05.683985 9341 tablet_bootstrap.cc:492] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:05.684437 9341 ts_tablet_manager.cc:1403] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.738s user 0.005s sys 0.005s
I20260926 01:56:05.685004 10352 raft_consensus.cc:2804] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:05.686662 9006 leader_election.cc:304] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:05.687404 10314 raft_consensus.cc:2749] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:05.695124 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 129752us
I20260926 01:56:05.699766 9341 raft_consensus.cc:359] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:56:05.701053 9003 leader_election.cc:343] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: b659eca3c97144aaacd41e747aecd96d
I20260926 01:56:05.710124 10352 raft_consensus.cc:697] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:05.731083 10352 consensus_queue.cc:237] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:05.692873 9192 consensus_meta.cc:287] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.536s user 0.010s sys 0.000s
W20260926 01:56:05.725932 9187 consensus_meta.cc:287] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532: Time spent flushing consensus metadata: real 0.537s user 0.009s sys 0.000s
I20260926 01:56:05.740209 9341 raft_consensus.cc:385] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:05.743405 9341 raft_consensus.cc:740] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:05.745175 10353 tablet.cc:1751] T 8a56044249bb4aaa81903d2a8aeb008e P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:05.713980 10191 raft_consensus.cc:515] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.667158 10340 consensus_queue.cc:1048] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.082s
I20260926 01:56:05.750087 10296 catalog_manager.cc:1623] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: acquiring CA information for follower catalog manager: success
I20260926 01:56:05.746188 9340 raft_consensus.cc:359] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.751303 9340 raft_consensus.cc:385] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:05.751638 9340 raft_consensus.cc:740] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:05.752256 9340 consensus_queue.cc:260] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.751796 10327 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:05.655577 10318 consensus_queue.cc:1048] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.093s
W20260926 01:56:05.759171 8874 consensus_meta.cc:287] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510: Time spent flushing consensus metadata: real 0.701s user 0.010s sys 0.002s
I20260926 01:56:05.764810 8624 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 6 FOLLOWER]: Rejecting Update request from peer ca25983598bc4c0688340d8e7bc0f6ba for earlier term 5. Current term is 6. Ops: [5.352-5.352]
I20260926 01:56:05.768893 9340 ts_tablet_manager.cc:1434] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.235s user 0.007s sys 0.000s
I20260926 01:56:05.769894 9340 tablet_bootstrap.cc:492] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:05.772138 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 127701us
I20260926 01:56:05.775632 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fdb7a7328c54031bd3afac1636d0f75" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:05.776238 9070 raft_consensus.cc:3060] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:05.782819 10338 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: INVALID_TERM, Last received: 5.351, Next index: 352, Last known committed idx: 350, Time since last communication: 0.001s
I20260926 01:56:05.785264 10246 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 5 LEADER]: Stepping down as leader of term 5
I20260926 01:56:05.785782 10246 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:56:05.786111 10174 consensus_queue.cc:1048] T 4623456312284a698ae69d4054909647 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:05.786823 10246 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 5.352, Last appended by leader: 352, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:05.787861 10246 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 5 FOLLOWER]: Advancing to term 6
I20260926 01:56:05.796296 8549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 351 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
I20260926 01:56:05.797037 9340 tablet_bootstrap.cc:654] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:05.750712 9341 consensus_queue.cc:260] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:05.802764 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 351 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849"
I20260926 01:56:05.804373 9985 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 5 FOLLOWER]: Advancing to term 6
I20260926 01:56:05.811312 9341 ts_tablet_manager.cc:1434] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.127s user 0.006s sys 0.003s
I20260926 01:56:05.813409 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef98e01cb9a44408ebf207163f208b5 in term 6.
I20260926 01:56:05.815207 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters:
I20260926 01:56:05.816252 10358 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 6 FOLLOWER]: Leader election won for term 6
I20260926 01:56:05.821131 8532 catalog_manager.cc:5838] T 56b0e40e5e964b90bd2dd158bbc4497a P 49bd320bf73f4252804ec867ec077510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } }
I20260926 01:56:05.823310 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]). Expected version 1 got 0
I20260926 01:56:05.824196 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:05.824486 10327 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 6 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:05.825135 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:05.827107 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:05.827951 8532 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:05.828310 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fdb7a7328c54031bd3afac1636d0f75" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
E20260926 01:56:05.826272 8794 catalog_manager.cc:839] Error processing pending assignments: Illegal state: error persisting updated tablet metadata: replica ca25983598bc4c0688340d8e7bc0f6ba is not leader of this config: current role FOLLOWER
I20260926 01:56:05.828949 9210 raft_consensus.cc:3060] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:05.831662 10191 leader_election.cc:290] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
E20260926 01:56:05.833267 8532 catalog_manager.cc:6001] Error updating tablets from 49bd320bf73f4252804ec867ec077510: Illegal state: replica ca25983598bc4c0688340d8e7bc0f6ba is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "17b8b0d54f5645e5963f1de217f94c84" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "04e79a2e52fb40619b2bc99396abda5b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 4 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "943a990fa10948e8961c02e9999569f5" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "72f666b1c069435b8ff0a3fae9f1e873" state: STOPPED consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_DELETED } updated_tablets { tablet_id: "f2ee16b0c3e84936abba8248f194786a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e53c9eb8a6d648b2a9359f02b7f8764b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "56b0e40e5e964b90bd2dd158bbc4497a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "1743e863cdd84f3f8ca413fa81cd1914" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "c9f54f648a55404fb6622b35d84368ee" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f1f020590a8b428788e2bb516a11cc45" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ee418347ebb04927913810d539ef57ea" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a09351c3db384b4383ca5ff0eaca982c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "562604137d484e83a51de2f778d58fca" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 4 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27759 live_row_count: 0 } } updated_tablets { tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } removed_tablet_ids: "9bac55da5b164c0f981914d39c87c33e" removed_tablet_ids: "3571967749844fd7859cf179a93cf403" removed_tablet_ids: "70ef5265eb6244aa96c8195aa30cdfb0" removed_tablet_ids: "1cba678b66744dcfb4d13ae07ebec33f" removed_tablet_ids: "3bf2fa5fbe4146b5adb27ff09de878d3" removed_tablet_ids: "c36ea09cf15647c0a344c9ba9b289d61" removed_tablet_ids: "ff004cf835c34095a836ad683cec9256" removed_tablet_ids: "39cd9130640d4b14beef42e9f1b36b99" removed_tablet_ids: "c27f1dd7bb0449a6a3fa371a8636e09b" removed_tablet_ids: "55452e70a7fe48efbafdc7cbb29c6272" removed_tablet_ids: "03086ec5451743fdbf2d5726d43f31b1" removed_tablet_ids: "b659eca3c97144aaacd41e747aecd96d" sequence_number: 47
I20260926 01:56:05.843459 9341 tablet_bootstrap.cc:492] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:05.843372 8532 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 48) took 3100 ms (client timeout 1000 ms). Trace:
W20260926 01:56:05.843938 8532 rpcz_store.cc:269] 0926 01:56:02.743228 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.387364 (+644136us) service_pool.cc:224] Handling call
0926 01:56:03.530134 (+142770us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:05.832501 (+2302367us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:05.832523 (+ 22us) write_op.cc:454] Released schema lock
0926 01:56:05.843190 (+ 10667us) inbound_call.cc:177] Queueing failure response
Related trace 'op':
0926 01:56:05.832301 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica ca25983598bc4c0688340d8e7bc0f6ba is not leader of this config: current role FOLLOWER)
0926 01:56:05.832337 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:05.832358 (+ 21us) write_op.cc:454] Released schema lock
0926 01:56:05.832380 (+ 22us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":22,"spinlock_wait_cycles":319275776,"child_traces":[["op",{}]]}
I20260926 01:56:05.857595 9210 raft_consensus.cc:2468] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
W20260926 01:56:05.866086 9006 leader_election.cc:343] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: b659eca3c97144aaacd41e747aecd96d
I20260926 01:56:05.867265 9006 leader_election.cc:304] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:05.872656 9341 tablet_bootstrap.cc:654] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:05.873334 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:05.877537 8866 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:58210 (request call id 54) took 1163 ms. Trace:
I20260926 01:56:05.877959 8866 rpcz_store.cc:276] 0926 01:56:04.713805 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:04.713920 (+ 115us) service_pool.cc:224] Handling call
0926 01:56:05.753336 (+1039416us) spinlock_profiling.cc:243] Waited 8.7 ms on lock 0x7b5c00040f8c. stack: 00007f94aec19323 00007f94aec19220 00007f94ae7bd574 0000000000516964 00000000005168f9 00007f94b616d1a1 00007f94b6312fd9 00007f94b61f95b0 00007f94b6e18960 00007f94b5568104 00007f94b5568092 00007f94b5567fc1 00007f94b5567f3c 00007f94b55671f2 00007f94b3a7c09c 00007f94b3a7b4d6
0926 01:56:05.877261 (+123925us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20017536}
I20260926 01:56:05.903915 9339 raft_consensus.cc:359] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.904614 9339 raft_consensus.cc:385] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:05.904879 9339 raft_consensus.cc:740] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:05.905462 9339 consensus_queue.cc:260] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.836753 10365 tablet.cc:1751] T 8a56044249bb4aaa81903d2a8aeb008e P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:05.915056 9070 raft_consensus.cc:2468] T 0fdb7a7328c54031bd3afac1636d0f75 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:05.917595 9341 tablet_bootstrap.cc:492] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:05.919276 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet f2ee16b0c3e84936abba8248f194786a (table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c]). Expected version 1 got 0
I20260926 01:56:05.920241 8533 catalog_manager.cc:5838] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } }
I20260926 01:56:05.921772 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]). Expected version 1 got 0
I20260926 01:56:05.922273 9341 ts_tablet_manager.cc:1403] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.007s
I20260926 01:56:05.926132 10358 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 6 LEADER]: Becoming Leader. State: Replica: 9ef98e01cb9a44408ebf207163f208b5, State: Running, Role: LEADER
I20260926 01:56:05.927224 10358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 5.351, Last appended by leader: 351, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:05.936334 9340 tablet_bootstrap.cc:492] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:05.936934 9340 ts_tablet_manager.cc:1403] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.167s user 0.007s sys 0.005s
I20260926 01:56:05.939589 9340 raft_consensus.cc:359] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.942209 10327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef98e01cb9a44408ebf207163f208b5. Latest consensus state: current_term: 6 leader_uuid: "9ef98e01cb9a44408ebf207163f208b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:05.946573 10327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:05.951109 10296 catalog_manager.cc:1651] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:56:05.925804 8549 raft_consensus.cc:2410] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ef98e01cb9a44408ebf207163f208b5 for term 6 because replica has last-logged OpId of term: 5 index: 352, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 351.
I20260926 01:56:05.970654 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:05.974175 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:05.975087 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:05.976994 8533 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]). Expected version 1 got 0
I20260926 01:56:05.979317 9341 raft_consensus.cc:359] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.979967 9341 raft_consensus.cc:385] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:05.980222 9341 raft_consensus.cc:740] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:05.980765 9341 consensus_queue.cc:260] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:05.982787 8526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260926 01:56:05.988623 9341 ts_tablet_manager.cc:1434] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
W20260926 01:56:05.992136 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 184184us
I20260926 01:56:06.003374 8532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 315) took 2285 ms. Trace:
I20260926 01:56:05.883412 10314 raft_consensus.cc:493] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:06.017783 9341 tablet_bootstrap.cc:492] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:06.019240 10335 log.cc:927] Time spent T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260926 01:56:06.021669 10314 raft_consensus.cc:515] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:05.994012 8843 leader_election.cc:304] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:06.001698 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f10cb68586b4a73abb86641b90c12f7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:06.023738 10314 leader_election.cc:290] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:06.015260 10315 raft_consensus.cc:2749] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:06.024696 10339 raft_consensus.cc:2804] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:06.025149 10339 raft_consensus.cc:697] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:06.025943 10339 consensus_queue.cc:237] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.030890 10381 tablet.cc:1751] T 229f4cdd841949e4a12e8555cdfe42cc P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:06.031350 10174 consensus_queue.cc:1048] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:06.036212 9340 raft_consensus.cc:385] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.036063 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f10cb68586b4a73abb86641b90c12f7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:06.036662 9070 raft_consensus.cc:3060] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.036705 9340 raft_consensus.cc:740] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.037452 9340 consensus_queue.cc:260] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.038317 8904 raft_consensus.cc:3060] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:06.039176 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 consensus state has no leader
I20260926 01:56:06.050012 8904 raft_consensus.cc:2468] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:06.057331 9152 leader_election.cc:304] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:06.058207 10377 raft_consensus.cc:2804] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:06.065511 10356 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51458 (request call id 253) took 1902 ms. Trace:
I20260926 01:56:06.071933 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 187) took 9973 ms. Trace:
I20260926 01:56:06.072466 9711 rpcz_store.cc:276] 0926 01:55:56.099565 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.161164 (+ 61599us) service_pool.cc:224] Handling call
0926 01:56:05.216461 (+9055297us) ts_tablet_manager.cc:1151] Deleting tablet 9bac55da5b164c0f981914d39c87c33e
0926 01:56:05.216899 (+ 438us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:05.287411 (+ 70512us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:05.666177 (+378766us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:06.071858 (+405681us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":341707,"tablet-delete.queue_time_us":8869718}
I20260926 01:56:06.085462 9340 ts_tablet_manager.cc:1434] T 1f10cb68586b4a73abb86641b90c12f7 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.148s user 0.005s sys 0.000s
I20260926 01:56:06.091002 9341 tablet_bootstrap.cc:654] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.093906 10377 raft_consensus.cc:697] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:06.094923 10377 consensus_queue.cc:237] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.097805 9711 tablet_replica.cc:333] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:06.098654 9711 raft_consensus.cc:2243] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:06.099567 9340 tablet_bootstrap.cc:492] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:06.101361 9339 ts_tablet_manager.cc:1434] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.491s user 0.013s sys 0.002s
I20260926 01:56:06.004315 8532 rpcz_store.cc:276] 0926 01:56:03.691506 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:05.973146 (+2281640us) service_pool.cc:224] Handling call
0926 01:56:05.976339 (+ 3193us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260926 01:56:06.106650 9275 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.190:44589)
I20260926 01:56:06.127302 9339 tablet_bootstrap.cc:492] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:06.128494 9711 raft_consensus.cc:2272] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:06.130376 9711 ts_tablet_manager.cc:1916] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:06.132679 10362 raft_consensus.cc:493] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:06.066020 10356 rpcz_store.cc:276] 0926 01:56:04.162891 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:04.163021 (+ 130us) service_pool.cc:224] Handling call
0926 01:56:06.065434 (+1902413us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:04.241433 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:56:04.241838 (+ 405us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:56:04.241868 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0926 01:56:04.241913 (+ 45us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:56:04.242046 (+ 133us) alter_schema_op.cc:128] START. Timestamp: 1790387764241973
0926 01:56:04.242064 (+ 18us) op_driver.cc:348] REPLICATION: starting
0926 01:56:04.242717 (+ 653us) log.cc:844] Serialized 173 byte log entry
0926 01:56:05.566669 (+1323952us) op_driver.cc:464] REPLICATION: finished
0926 01:56:05.638909 (+ 72240us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:56:05.640400 (+ 1491us) log.cc:844] Serialized 5 byte log entry
0926 01:56:06.063875 (+423475us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:06.064626 (+ 751us) log.cc:844] Serialized 14 byte log entry
0926 01:56:06.065271 (+ 645us) alter_schema_op.cc:73] Released schema lock
0926 01:56:06.065302 (+ 31us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":72033,"prepare.queue_time_us":77415,"prepare.run_cpu_time_us":79307,"prepare.run_wall_time_us":769758,"raft.queue_time_us":23398,"raft.run_cpu_time_us":1833,"raft.run_wall_time_us":1831,"replication_time_us":1324464,"thread_start_us":868103,"threads_started":4,"wal-append.queue_time_us":464331}]]}
I20260926 01:56:06.138939 10356 tablet.cc:1751] T 229f4cdd841949e4a12e8555cdfe42cc P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:06.133180 10362 raft_consensus.cc:515] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:06.167061 10362 leader_election.cc:290] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:06.168397 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e12827ad93c47cf9013e238215ceb7a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:06.170008 9152 leader_election.cc:343] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:06.176568 9340 tablet_bootstrap.cc:654] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.215940 9341 tablet_bootstrap.cc:492] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:06.220189 9340 tablet_bootstrap.cc:492] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:06.220651 9340 ts_tablet_manager.cc:1403] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.005s
I20260926 01:56:06.216864 9341 ts_tablet_manager.cc:1403] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.199s user 0.017s sys 0.005s
I20260926 01:56:06.228648 10186 ts_tablet_manager.cc:1929] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:06.229183 10186 log.cc:1199] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:06.228874 9341 raft_consensus.cc:359] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.229826 9341 raft_consensus.cc:385] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.230064 10186 ts_tablet_manager.cc:1950] T 1f6d9fedc7b241b59b25d591c7dc2fea P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:06.230140 9341 raft_consensus.cc:740] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.231070 9341 consensus_queue.cc:260] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.233428 9341 ts_tablet_manager.cc:1434] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:56:06.233881 10246 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:06.234382 10246 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:06.238904 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 352 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
E20260926 01:56:06.119753 8533 catalog_manager.cc:6001] Error updating tablets from bc00feec3f564608851f0112e97ab532: Illegal state: replica ca25983598bc4c0688340d8e7bc0f6ba is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "943a990fa10948e8961c02e9999569f5" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d0bb5090e4874ae188b6d1ffe8649b4e" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "681900e0fac74cdabaea1d8a7fb4a189" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f100eb5dc9a8452e943d4cce5284bc8a" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "28eaf7381f364a2682dd07f2bd90ede3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27506 live_row_count: 0 } } updated_tablets { tablet_id: "229f4cdd841949e4a12e8555cdfe42cc" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27506 live_row_count: 0 } } updated_tablets { tablet_id: "272bb87fe22c4720bcb3f0e07482dd8b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "3571967749844fd7859cf179a93cf403" state: STOPPED consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f2ee16b0c3e84936abba8248f194786a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "1f6d9fedc7b241b59b25d591c7dc2fea" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "8a56044249bb4aaa81903d2a8aeb008e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ac2e6e544aae4ac3a2d3cc116d22e581" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "1743e863cdd84f3f8ca413fa81cd1914" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "661148692f9e4127af9abf5cc6cc0f54" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "04e79a2e52fb40619b2bc99396abda5b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } } } schema_version: 4 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27759 live_row_count: 0 } } updated_tablets { tablet_id: "8dfff96cc29145e7a72213bb5ecca1ae" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27981 live_row_count: 0 } } updated_tablets { tablet_id: "562604137d484e83a51de2f778d58fca" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } } } schema_version: 4 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 28506 live_row_count: 0 } } updated_tablets { tablet_id: "a09351c3db384b4383ca5ff0eaca982c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f1f020590a8b428788e2bb516a11cc45" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27462 live_row_count: 0 } } updated_tablets { tablet_id: "ee418347ebb04927913810d539ef57ea" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "e53c9eb8a6d648b2a9359f02b7f8764b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6d101ad03f8e4d7f8bf70ea993c4f58b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "c9f54f648a55404fb6622b35d84368ee" state: RUNNING consensus_state { current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } removed_tablet_ids: "72f666b1c069435b8ff0a3fae9f1e873" removed_tablet_ids: "70ef5265eb6244aa96c8195aa30cdfb0" removed_tablet_ids: "39cd9130640d4b14beef42e9f1b36b99" removed_tablet_ids: "c27f1dd7bb0449a6a3fa371a8636e09b" removed_tablet_ids: "03086ec5451743fdbf2d5726d43f31b1" removed_tablet_ids: "55452e70a7fe48efbafdc7cbb29c6272" removed_tablet_ids: "75d4598596ff49378fc3536fff0ac1b1" removed_tablet_ids: "f3bb1643da054583b97efd2e1a9e9c2f" removed_tablet_ids: "23d2805b41d645d3bb905f26a380f383" removed_tablet_ids: "b659eca3c97144aaacd41e747aecd96d" removed_tablet_ids: "1866cd97bb024a9baecf101a6d5b1ee4" removed_tablet_ids: "1cba678b66744dcfb4d13ae07ebec33f" removed_tablet_ids: "c36ea09cf15647c0a344c9ba9b289d61" removed_tablet_ids: "1df1d4daaddc4ac4aa02aa7763ea3184" removed_tablet_ids: "32e0fe60b8744dbb8ae2bafefe3e834d" removed_tablet_ids: "ff004cf835c34095a836ad683cec9256" removed_tablet_ids: "dfaca3fc02634f459083d23f3102f725" removed_tablet_ids: "9bac55da5b164c0f981914d39c87c33e" removed_tablet_ids: "3bf2fa5fbe4146b5adb27ff09de878d3" removed_tablet_ids: "01052a5e22a74a19919bae9dd0296291" sequence_number: 35
W20260926 01:56:06.240684 8533 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.131:48005 (request call id 36) took 3250 ms (client timeout 1000 ms). Trace:
W20260926 01:56:06.241040 8533 rpcz_store.cc:269] 0926 01:56:02.990161 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.461725 (+471564us) service_pool.cc:224] Handling call
0926 01:56:03.538698 (+ 76973us) spinlock_profiling.cc:243] Waited 71.9 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:06.119507 (+2580809us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:06.119529 (+ 22us) write_op.cc:454] Released schema lock
0926 01:56:06.240591 (+121062us) inbound_call.cc:177] Queueing failure response
Related trace 'op':
0926 01:56:06.119297 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica ca25983598bc4c0688340d8e7bc0f6ba is not leader of this config: current role FOLLOWER)
0926 01:56:06.119335 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:06.119354 (+ 19us) write_op.cc:454] Released schema lock
0926 01:56:06.119372 (+ 18us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":23,"spinlock_wait_cycles":165291392,"child_traces":[["op",{}]]}
I20260926 01:56:06.243201 10246 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:06.265566 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 278) took 4611 ms. Trace:
I20260926 01:56:06.267263 9341 tablet_bootstrap.cc:492] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:06.270536 9340 raft_consensus.cc:359] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:06.271461 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 352 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:06.272106 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 6.
I20260926 01:56:06.272607 9340 raft_consensus.cc:385] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.272949 9340 raft_consensus.cc:740] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.273514 8524 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5
I20260926 01:56:06.265991 10186 rpcz_store.cc:276] 0926 01:56:01.653805 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.653902 (+ 97us) service_pool.cc:224] Handling call
0926 01:56:05.321254 (+3667352us) ts_tablet_manager.cc:1151] Deleting tablet 1f6d9fedc7b241b59b25d591c7dc2fea
0926 01:56:05.321313 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:05.996566 (+675253us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:06.188910 (+192344us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:06.265478 (+ 76568us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3633960}
I20260926 01:56:06.274736 10246 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260926 01:56:06.275040 10246 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:06.275372 10246 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 6 FOLLOWER]: Advancing to term 7
I20260926 01:56:06.275700 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 1f6d9fedc7b241b59b25d591c7dc2fea (table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]) successfully deleted
I20260926 01:56:06.277969 9341 tablet_bootstrap.cc:654] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.288120 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 279) took 4633 ms. Trace:
I20260926 01:56:06.288509 10186 rpcz_store.cc:276] 0926 01:56:01.654551 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.654661 (+ 110us) service_pool.cc:224] Handling call
0926 01:56:06.277866 (+4623205us) ts_tablet_manager.cc:1151] Deleting tablet 8d198e51b88948548cfd95dd893a26ef
0926 01:56:06.287619 (+ 9753us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:06.288041 (+ 422us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1792,"tablet-delete.queue_time_us":4590429}
I20260926 01:56:06.125229 9070 raft_consensus.cc:2468] T 1f10cb68586b4a73abb86641b90c12f7 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:56:06.293594 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 8d198e51b88948548cfd95dd893a26ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d198e51b88948548cfd95dd893a26ef already in progress: creating tablet
I20260926 01:56:06.295032 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 281) took 3711 ms. Trace:
I20260926 01:56:06.295401 10186 rpcz_store.cc:276] 0926 01:56:02.583719 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:02.610990 (+ 27271us) service_pool.cc:224] Handling call
0926 01:56:06.294564 (+3683574us) ts_tablet_manager.cc:1151] Deleting tablet bcb000f9234d4382ad019f63015b92e7
0926 01:56:06.294623 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:06.294964 (+ 341us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3673242}
W20260926 01:56:06.296840 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet bcb000f9234d4382ad019f63015b92e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcb000f9234d4382ad019f63015b92e7 already in progress: creating tablet
I20260926 01:56:06.299032 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 283) took 2943 ms. Trace:
I20260926 01:56:06.299400 10186 rpcz_store.cc:276] 0926 01:56:03.355715 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.355897 (+ 182us) service_pool.cc:224] Handling call
0926 01:56:06.298593 (+2942696us) ts_tablet_manager.cc:1151] Deleting tablet b612e8955d5f42dd8658320031fe36ca
0926 01:56:06.298642 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:06.298966 (+ 324us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2882489}
W20260926 01:56:06.300630 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 410791us
W20260926 01:56:06.300858 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet b612e8955d5f42dd8658320031fe36ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b612e8955d5f42dd8658320031fe36ca already in progress: creating tablet
I20260926 01:56:06.302167 9339 tablet_bootstrap.cc:654] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.307257 10186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42408 (request call id 285) took 2242 ms. Trace:
I20260926 01:56:06.307649 10186 rpcz_store.cc:276] 0926 01:56:04.064816 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:04.064966 (+ 150us) service_pool.cc:224] Handling call
0926 01:56:06.306801 (+2241835us) ts_tablet_manager.cc:1151] Deleting tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e
0926 01:56:06.306853 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:06.307184 (+ 331us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2241248}
I20260926 01:56:06.315531 10246 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
W20260926 01:56:06.315557 8527 catalog_manager.cc:5043] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): delete failed for tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e already in progress: creating tablet
I20260926 01:56:06.317809 10246 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 7 election: Requested vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:06.319108 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 352 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849"
I20260926 01:56:06.319187 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 7 candidate_status { last_received { term: 5 index: 352 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:56:06.319706 9985 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 6 FOLLOWER]: Advancing to term 7
I20260926 01:56:06.327535 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c8761a6c3f7435193a91bfc4fe7fa86" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:06.328191 9218 raft_consensus.cc:2468] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:06.332141 10355 raft_consensus.cc:493] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:06.333066 10355 raft_consensus.cc:515] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:06.335911 10355 leader_election.cc:290] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:06.337877 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 7.
I20260926 01:56:06.344632 9340 consensus_queue.cc:260] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:06.347946 8549 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 7 FOLLOWER]: Rejecting Update request from peer 9ef98e01cb9a44408ebf207163f208b5 for earlier term 6. Current term is 7. Ops: []
I20260926 01:56:06.349925 10327 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 }, Status: INVALID_TERM, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260926 01:56:06.350718 10327 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [LEADER]: Peer ca25983598bc4c0688340d8e7bc0f6ba log is divergent from this leader: its last log entry 5.352 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 351
I20260926 01:56:06.351148 10327 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260926 01:56:06.351445 10327 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9ef98e01cb9a44408ebf207163f208b5, State: Running, Role: LEADER
I20260926 01:56:06.352208 10327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 6.352, Last appended by leader: 352, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:06.353339 10327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 6 FOLLOWER]: Advancing to term 7
I20260926 01:56:06.353578 8524 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5
I20260926 01:56:06.360221 10246 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 7 FOLLOWER]: Leader election won for term 7
I20260926 01:56:06.364789 9341 tablet_bootstrap.cc:492] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:06.365280 9341 ts_tablet_manager.cc:1403] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.008s
I20260926 01:56:06.368091 9341 raft_consensus.cc:359] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.368777 9341 raft_consensus.cc:385] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.369040 9341 raft_consensus.cc:740] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.371177 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:06.373744 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:06.376883 10246 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 7 LEADER]: Becoming Leader. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:56:06.378018 10246 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 5.352, Last appended by leader: 352, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:06.378811 9341 consensus_queue.cc:260] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.378608 8534 catalog_manager.cc:5838] T 56b0e40e5e964b90bd2dd158bbc4497a P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } }
I20260926 01:56:06.381209 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 1f10cb68586b4a73abb86641b90c12f7 (table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f]). Expected version 1 got 0
I20260926 01:56:06.347649 9340 ts_tablet_manager.cc:1434] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.127s user 0.007s sys 0.000s
I20260926 01:56:06.385991 9340 tablet_bootstrap.cc:492] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:06.388725 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:06.389766 8534 catalog_manager.cc:5838] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } }
I20260926 01:56:06.392040 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]). Expected version 1 got 0
I20260926 01:56:06.392025 10397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 7 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:06.392942 10397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:06.393240 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 9bac55da5b164c0f981914d39c87c33e (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f]) successfully deleted
I20260926 01:56:06.393409 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 880d77d42c394e6a9cf0b3768a915402 (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]). Expected version 2 got 0
I20260926 01:56:06.393935 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 4623456312284a698ae69d4054909647 (table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa]). Expected version 1 got 0
I20260926 01:56:06.395061 8534 catalog_manager.cc:5838] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: UNKNOWN } } }
I20260926 01:56:06.399557 9341 ts_tablet_manager.cc:1434] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260926 01:56:06.400549 9341 tablet_bootstrap.cc:492] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:06.397869 8534 catalog_manager.cc:5838] T 661148692f9e4127af9abf5cc6cc0f54 P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } }
I20260926 01:56:06.402535 9070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e12827ad93c47cf9013e238215ceb7a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:06.403339 9070 raft_consensus.cc:2468] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:56:06.403422 8534 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:06.417290 9341 tablet_bootstrap.cc:654] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.417768 10339 raft_consensus.cc:493] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:06.420257 10355 raft_consensus.cc:493] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:06.420800 10355 raft_consensus.cc:515] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.418237 10339 raft_consensus.cc:515] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.425109 10339 leader_election.cc:290] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:06.429188 9006 leader_election.cc:304] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:06.430850 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d068e93fd374a6091d7473db069a7fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:06.433117 8843 leader_election.cc:343] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:06.449718 9155 leader_election.cc:304] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510
I20260926 01:56:06.451574 10192 raft_consensus.cc:2804] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:06.451965 10192 raft_consensus.cc:493] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:06.452282 10192 raft_consensus.cc:3060] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.463548 8532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 316) took 2768 ms. Trace:
I20260926 01:56:06.463980 8532 rpcz_store.cc:276] 0926 01:56:03.695061 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:06.462892 (+2767831us) service_pool.cc:224] Handling call
0926 01:56:06.463459 (+ 567us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:06.463879 10192 raft_consensus.cc:515] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:06.465252 8532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 317) took 2359 ms. Trace:
I20260926 01:56:06.465557 8532 rpcz_store.cc:276] 0926 01:56:04.105512 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:06.464706 (+2359194us) service_pool.cc:224] Handling call
0926 01:56:06.465192 (+ 486us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:06.468012 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e12827ad93c47cf9013e238215ceb7a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:06.468607 8904 raft_consensus.cc:3060] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.476027 8904 raft_consensus.cc:2468] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:06.476991 10355 leader_election.cc:290] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:06.477303 9152 leader_election.cc:304] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:06.478194 10362 raft_consensus.cc:2804] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:06.478384 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e12827ad93c47cf9013e238215ceb7a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:06.478677 10362 raft_consensus.cc:697] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:06.478972 9067 raft_consensus.cc:3060] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.479403 10362 consensus_queue.cc:237] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:56:06.493697 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 64639us
I20260926 01:56:06.494083 10192 leader_election.cc:290] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:06.495024 10395 raft_consensus.cc:2804] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:06.495430 10395 raft_consensus.cc:493] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:06.495736 10395 raft_consensus.cc:3060] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.466372 8532 master_service.cc:960] ReplaceTablet: received request to replace tablet 030f4d683e404fbd8d2e6863e18f71c0 from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:06.503805 8532 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:42048 (request call id 318) took 2290 ms. Trace:
I20260926 01:56:06.504184 8532 rpcz_store.cc:276] 0926 01:56:04.213500 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:06.466215 (+2252715us) service_pool.cc:224] Handling call
0926 01:56:06.503460 (+ 37245us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60a0e1 00007f717c1d2d37 00007f717c1d2cc2 00007f717c1d2bf1 00007f717c1d2b6c 00007f717c1d1e22 00007f717a45c09c 00007f717a45b4d6
0926 01:56:06.503698 (+ 238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8974208}
I20260926 01:56:06.505796 8532 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 319) took 2232 ms. Trace:
W20260926 01:56:06.506258 9275 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.76.190:44589)
I20260926 01:56:06.509284 9985 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 5 index: 283. Preceding OpId from leader: term: 7 index: 354. (index mismatch)
I20260926 01:56:06.511026 10397 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 281, Time since last communication: 0.000s
W20260926 01:56:06.512451 9006 leader_election.cc:343] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:06.517638 10362 raft_consensus.cc:493] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:06.518292 10362 raft_consensus.cc:515] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.521210 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:06.526779 10362 leader_election.cc:290] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:06.528762 8843 consensus_peers.cc:597] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:06.532069 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbf222f9e14d4332b679f7ee94af1e0e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:06.532860 8624 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 6 index: 352. Preceding OpId from leader: term: 7 index: 354. (index mismatch)
I20260926 01:56:06.533023 9072 raft_consensus.cc:2393] T dbf222f9e14d4332b679f7ee94af1e0e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:06.534775 9155 leader_election.cc:304] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:06.535766 10246 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 350, Time since last communication: 0.000s
I20260926 01:56:06.536409 10246 consensus_queue.cc:1243] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Peer 9ef98e01cb9a44408ebf207163f208b5 log is divergent from this leader: its last log entry 6.352 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 350
I20260926 01:56:06.537106 10362 raft_consensus.cc:3060] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.539531 8624 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.350->[5.351-7.354] Dedup: 5.351->[5.352-7.354]
I20260926 01:56:06.540017 8624 pending_rounds.cc:85] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Aborting all ops after (but not including) 351
I20260926 01:56:06.540228 8624 pending_rounds.cc:107] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Aborting uncommitted NO_OP operation due to leader change: 6.352
I20260926 01:56:06.540501 8624 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260926 01:56:06.543608 9006 leader_election.cc:343] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: cbf8818e65d74a32ba21dcb8a9a56a62
I20260926 01:56:06.544456 9340 tablet_bootstrap.cc:654] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.546139 8532 rpcz_store.cc:276] 0926 01:56:04.273298 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:06.505233 (+2231935us) service_pool.cc:224] Handling call
0926 01:56:06.505713 (+ 480us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:06.550519 9067 raft_consensus.cc:2468] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
W20260926 01:56:06.551894 9275 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.76.190:44589)
W20260926 01:56:06.564429 9155 consensus_peers.cc:597] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16f49a5a10ea4f7a821d89f18e3af560. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:06.568504 8533 master_service.cc:960] ReplaceTablet: received request to replace tablet 880d77d42c394e6a9cf0b3768a915402 from {username='slave'} at 127.0.0.1:42048
W20260926 01:56:06.571437 8526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260926 01:56:06.573462 10362 raft_consensus.cc:2749] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:06.579882 10340 consensus_queue.cc:1048] T e18ddaa6f0e748cf930e29cbca3844b1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:06.591018 9341 tablet_bootstrap.cc:492] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:06.591493 9341 ts_tablet_manager.cc:1403] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.191s user 0.008s sys 0.003s
I20260926 01:56:06.608368 9340 tablet_bootstrap.cc:492] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:06.609684 8498 master-stress-test.cc:579] Killing test thread 1/10
I20260926 01:56:06.612869 9340 ts_tablet_manager.cc:1403] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.227s user 0.012s sys 0.000s
I20260926 01:56:06.615703 9340 raft_consensus.cc:359] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.616643 9340 raft_consensus.cc:385] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.616967 9340 raft_consensus.cc:740] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.617756 9340 consensus_queue.cc:260] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.622314 8533 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:44110 (request call id 23) took 3065 ms. Trace:
I20260926 01:56:06.622680 8533 rpcz_store.cc:276] 0926 01:56:03.556515 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:06.601914 (+3045399us) service_pool.cc:224] Handling call
0926 01:56:06.622223 (+ 20309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:06.622769 9341 raft_consensus.cc:359] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.623813 9341 raft_consensus.cc:385] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.624114 9341 raft_consensus.cc:740] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.624926 9341 consensus_queue.cc:260] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.626016 8533 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:49078 (request call id 1) took 1651 ms. Trace:
I20260926 01:56:06.626338 8533 rpcz_store.cc:276] 0926 01:56:04.975213 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:06.625483 (+1650270us) service_pool.cc:224] Handling call
0926 01:56:06.625948 (+ 465us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:06.639966 9341 ts_tablet_manager.cc:1434] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260926 01:56:06.641068 9341 tablet_bootstrap.cc:492] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:06.641402 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c8761a6c3f7435193a91bfc4fe7fa86" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:06.552881 10395 raft_consensus.cc:515] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.642055 8905 raft_consensus.cc:2468] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:06.643986 10395 leader_election.cc:290] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:06.568634 9339 tablet_bootstrap.cc:492] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:06.646847 9339 ts_tablet_manager.cc:1403] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.520s user 0.016s sys 0.007s
I20260926 01:56:06.570765 9711 ts_tablet_manager.cc:1929] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:06.651674 9711 log.cc:1199] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/6d56f3a5dfed4ac78f1da92e22db558a
I20260926 01:56:06.651801 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c8761a6c3f7435193a91bfc4fe7fa86" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:06.652531 9212 raft_consensus.cc:3060] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.652662 9711 ts_tablet_manager.cc:1950] T 6d56f3a5dfed4ac78f1da92e22db558a P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:06.571914 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d068e93fd374a6091d7473db069a7fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:06.653519 9072 raft_consensus.cc:2468] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:06.618180 10127 raft_consensus.cc:493] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:06.654762 10127 raft_consensus.cc:515] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.656888 10127 leader_election.cc:290] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:06.658752 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:06.660424 9212 raft_consensus.cc:2468] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:06.660972 9340 ts_tablet_manager.cc:1434] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260926 01:56:06.661998 9340 tablet_bootstrap.cc:492] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:06.662817 9006 leader_election.cc:343] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:06.663481 9006 leader_election.cc:304] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:06.664460 10127 raft_consensus.cc:2804] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:06.664896 10127 raft_consensus.cc:697] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:06.666260 10127 consensus_queue.cc:237] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.649828 9339 raft_consensus.cc:359] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:06.671758 9339 raft_consensus.cc:385] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.672039 9339 raft_consensus.cc:740] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.672685 9339 consensus_queue.cc:260] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:06.674918 9339 ts_tablet_manager.cc:1434] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260926 01:56:06.676008 9339 tablet_bootstrap.cc:492] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:06.680603 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 197) took 9696 ms. Trace:
I20260926 01:56:06.681457 9341 tablet_bootstrap.cc:654] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.680991 9711 rpcz_store.cc:276] 0926 01:55:56.984443 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:56.984548 (+ 105us) service_pool.cc:224] Handling call
0926 01:56:06.097620 (+9113072us) ts_tablet_manager.cc:1151] Deleting tablet 6d56f3a5dfed4ac78f1da92e22db558a
0926 01:56:06.097674 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:06.403729 (+306055us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:06.570491 (+166762us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:06.680508 (+110017us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29323,"tablet-delete.queue_time_us":9112489}
I20260926 01:56:06.682742 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 6d56f3a5dfed4ac78f1da92e22db558a (table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]) successfully deleted
I20260926 01:56:06.682997 9711 tablet_replica.cc:333] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:06.710906 9711 raft_consensus.cc:2243] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:06.711527 9711 raft_consensus.cc:2272] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:06.714676 9711 ts_tablet_manager.cc:1916] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:06.715808 9340 tablet_bootstrap.cc:654] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.719079 9339 tablet_bootstrap.cc:654] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:04.324942 9990 rpcz_store.cc:276] 0926 01:55:58.373070 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.373268 (+ 198us) service_pool.cc:224] Handling call
0926 01:56:00.151644 (+1778376us) raft_consensus.cc:1471] Updating replica for 276 ops
0926 01:56:00.313737 (+162093us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.313767 (+ 30us) write_op.cc:454] Released schema lock
0926 01:56:00.315989 (+ 2222us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:00.316016 (+ 27us) write_op.cc:454] Released schema lock
0926 01:56:00.317331 (+ 1315us) raft_consensus.cc:1534] Early marking committed up to index 7
0926 01:56:00.408510 (+ 91179us) raft_consensus.cc:1539] Triggering prepare for 276 ops
0926 01:56:00.409278 (+ 768us) write_op.cc:270] Start()
0926 01:56:00.409425 (+ 147us) write_op.cc:276] Timestamp: P: 1790387740554979 usec, L: 0
0926 01:56:00.412851 (+ 3426us) write_op.cc:270] Start()
0926 01:56:00.412963 (+ 112us) write_op.cc:276] Timestamp: P: 1790387740876860 usec, L: 0
0926 01:56:00.413580 (+ 617us) write_op.cc:270] Start()
0926 01:56:00.413670 (+ 90us) write_op.cc:276] Timestamp: P: 1790387740879499 usec, L: 0
0926 01:56:00.414239 (+ 569us) write_op.cc:270] Start()
0926 01:56:00.414326 (+ 87us) write_op.cc:276] Timestamp: P: 1790387740899511 usec, L: 0
0926 01:56:00.414956 (+ 630us) write_op.cc:270] Start()
0926 01:56:00.415068 (+ 112us) write_op.cc:276] Timestamp: P: 1790387740936076 usec, L: 0
0926 01:56:00.415571 (+ 503us) write_op.cc:270] Start()
0926 01:56:00.415658 (+ 87us) write_op.cc:276] Timestamp: P: 1790387740942938 usec, L: 0
0926 01:56:00.416163 (+ 505us) write_op.cc:270] Start()
0926 01:56:00.416242 (+ 79us) write_op.cc:276] Timestamp: P: 1790387740960365 usec, L: 0
0926 01:56:00.441200 (+ 24958us) write_op.cc:270] Start()
0926 01:56:00.441311 (+ 111us) write_op.cc:276] Timestamp: P: 1790387740979384 usec, L: 0
0926 01:56:00.441928 (+ 617us) write_op.cc:270] Start()
0926 01:56:00.442012 (+ 84us) write_op.cc:276] Timestamp: P: 1790387740984470 usec, L: 0
0926 01:56:00.442509 (+ 497us) write_op.cc:270] Start()
0926 01:56:00.443044 (+ 535us) write_op.cc:276] Timestamp: P: 1790387740989511 usec, L: 0
0926 01:56:00.443535 (+ 491us) write_op.cc:270] Start()
0926 01:56:00.443636 (+ 101us) write_op.cc:276] Timestamp: P: 1790387741006117 usec, L: 0
0926 01:56:00.444151 (+ 515us) write_op.cc:270] Start()
0926 01:56:00.444231 (+ 80us) write_op.cc:276] Timestamp: P: 1790387741060815 usec, L: 0
0926 01:56:00.444697 (+ 466us) write_op.cc:270] Start()
0926 01:56:00.444770 (+ 73us) write_op.cc:276] Timestamp: P: 1790387741113918 usec, L: 0
0926 01:56:00.502421 (+ 57651us) write_op.cc:270] Start()
0926 01:56:00.502563 (+ 142us) write_op.cc:276] Timestamp: P: 1790387741134176 usec, L: 0
0926 01:56:00.503288 (+ 725us) write_op.cc:270] Start()
0926 01:56:00.503372 (+ 84us) write_op.cc:276] Timestamp: P: 1790387741145341 usec, L: 0
0926 01:56:00.547219 (+ 43847us) write_op.cc:270] Start()
0926 01:56:00.547338 (+ 119us) write_op.cc:276] Timestamp: P: 1790387741203722 usec, L: 0
0926 01:56:00.547953 (+ 615us) write_op.cc:270] Start()
0926 01:56:00.548031 (+ 78us) write_op.cc:276] Timestamp: P: 1790387741210043 usec, L: 0
0926 01:56:00.548488 (+ 457us) write_op.cc:270] Start()
0926 01:56:00.548567 (+ 79us) write_op.cc:276] Timestamp: P: 1790387741213363 usec, L: 0
0926 01:56:00.564495 (+ 15928us) write_op.cc:270] Start()
0926 01:56:00.564621 (+ 126us) write_op.cc:276] Timestamp: P: 1790387741233535 usec, L: 0
0926 01:56:00.565101 (+ 480us) write_op.cc:270] Start()
0926 01:56:00.565181 (+ 80us) write_op.cc:276] Timestamp: P: 1790387741237912 usec, L: 0
0926 01:56:00.590635 (+ 25454us) write_op.cc:270] Start()
0926 01:56:00.590784 (+ 149us) write_op.cc:276] Timestamp: P: 1790387741260542 usec, L: 0
0926 01:56:00.591489 (+ 705us) write_op.cc:270] Start()
0926 01:56:00.591718 (+ 229us) write_op.cc:276] Timestamp: P: 1790387741299343 usec, L: 0
0926 01:56:00.592249 (+ 531us) write_op.cc:270] Start()
0926 01:56:00.592333 (+ 84us) write_op.cc:276] Timestamp: P: 1790387741374174 usec, L: 0
0926 01:56:00.592828 (+ 495us) write_op.cc:270] Start()
0926 01:56:00.592946 (+ 118us) write_op.cc:276] Timestamp: P: 1790387741431676 usec, L: 0
0926 01:56:00.593446 (+ 500us) write_op.cc:270] Start()
0926 01:56:00.593742 (+ 296us) write_op.cc:276] Timestamp: P: 1790387741481427 usec, L: 0
0926 01:56:00.594230 (+ 488us) write_op.cc:270] Start()
0926 01:56:00.594309 (+ 79us) write_op.cc:276] Timestamp: P: 1790387741488766 usec, L: 0
0926 01:56:00.637513 (+ 43204us) write_op.cc:270] Start()
0926 01:56:00.637617 (+ 104us) write_op.cc:276] Timestamp: P: 1790387741525730 usec, L: 0
0926 01:56:00.638245 (+ 628us) write_op.cc:270] Start()
0926 01:56:00.638336 (+ 91us) write_op.cc:276] Timestamp: P: 1790387741530342 usec, L: 0
0926 01:56:00.639006 (+ 670us) write_op.cc:270] Start()
0926 01:56:00.639146 (+ 140us) write_op.cc:276] Timestamp: P: 1790387741534949 usec, L: 0
0926 01:56:00.639698 (+ 552us) write_op.cc:270] Start()
0926 01:56:00.639775 (+ 77us) write_op.cc:276] Timestamp: P: 1790387741696472 usec, L: 0
0926 01:56:00.640241 (+ 466us) write_op.cc:270] Start()
0926 01:56:00.640315 (+ 74us) write_op.cc:276] Timestamp: P: 1790387741720305 usec, L: 0
0926 01:56:00.640771 (+ 456us) write_op.cc:270] Start()
0926 01:56:00.640850 (+ 79us) write_op.cc:276] Timestamp: P: 1790387741736053 usec, L: 0
0926 01:56:00.641395 (+ 545us) write_op.cc:270] Start()
0926 01:56:00.641473 (+ 78us) write_op.cc:276] Timestamp: P: 1790387741781745 usec, L: 0
0926 01:56:00.641929 (+ 456us) write_op.cc:270] Start()
0926 01:56:00.642005 (+ 76us) write_op.cc:276] Timestamp: P: 1790387741833313 usec, L: 0
0926 01:56:00.783307 (+141302us) write_op.cc:270] Start()
0926 01:56:00.811242 (+ 27935us) write_op.cc:276] Timestamp: P: 1790387741838074 usec, L: 0
0926 01:56:00.812087 (+ 845us) write_op.cc:270] Start()
0926 01:56:00.812191 (+ 104us) write_op.cc:276] Timestamp: P: 1790387741863545 usec, L: 0
0926 01:56:00.812739 (+ 548us) write_op.cc:270] Start()
0926 01:56:00.812819 (+ 80us) write_op.cc:276] Timestamp: P: 1790387741867333 usec, L: 0
0926 01:56:00.999258 (+186439us) write_op.cc:270] Start()
0926 01:56:00.999381 (+ 123us) write_op.cc:276] Timestamp: P: 1790387741897447 usec, L: 0
0926 01:56:01.000163 (+ 782us) write_op.cc:270] Start()
0926 01:56:01.000254 (+ 91us) write_op.cc:276] Timestamp: P: 1790387741971876 usec, L: 0
0926 01:56:01.000882 (+ 628us) write_op.cc:270] Start()
0926 01:56:01.000964 (+ 82us) write_op.cc:276] Timestamp: P: 1790387742060651 usec, L: 0
0926 01:56:01.001475 (+ 511us) write_op.cc:270] Start()
0926 01:56:01.001560 (+ 85us) write_op.cc:276] Timestamp: P: 1790387742085893 usec, L: 0
0926 01:56:01.002020 (+ 460us) write_op.cc:270] Start()
0926 01:56:01.002129 (+ 109us) write_op.cc:276] Timestamp: P: 1790387742142291 usec, L: 0
0926 01:56:01.002659 (+ 530us) write_op.cc:270] Start()
0926 01:56:01.002747 (+ 88us) write_op.cc:276] Timestamp: P: 1790387742243505 usec, L: 0
0926 01:56:01.003209 (+ 462us) write_op.cc:270] Start()
0926 01:56:01.003286 (+ 77us) write_op.cc:276] Timestamp: P: 1790387742249576 usec, L: 0
0926 01:56:01.004669 (+ 1383us) write_op.cc:270] Start()
0926 01:56:01.004762 (+ 93us) write_op.cc:276] Timestamp: P: 1790387742253935 usec, L: 0
0926 01:56:01.005232 (+ 470us) write_op.cc:270] Start()
0926 01:56:01.005312 (+ 80us) write_op.cc:276] Timestamp: P: 1790387742303934 usec, L: 0
0926 01:56:01.005831 (+ 519us) write_op.cc:270] Start()
0926 01:56:01.005915 (+ 84us) write_op.cc:276] Timestamp: P: 1790387742375986 usec, L: 0
0926 01:56:01.730798 (+724883us) write_op.cc:270] Start()
0926 01:56:01.731000 (+ 202us) write_op.cc:276] Timestamp: P: 1790387742379667 usec, L: 0
0926 01:56:01.731763 (+ 763us) write_op.cc:270] Start()
0926 01:56:01.731854 (+ 91us) write_op.cc:276] Timestamp: P: 1790387742483430 usec, L: 0
0926 01:56:01.732385 (+ 531us) write_op.cc:270] Start()
0926 01:56:01.732466 (+ 81us) write_op.cc:276] Timestamp: P: 1790387742585906 usec, L: 0
0926 01:56:01.732946 (+ 480us) write_op.cc:270] Start()
0926 01:56:01.733045 (+ 99us) write_op.cc:276] Timestamp: P: 1790387742621725 usec, L: 0
0926 01:56:01.733532 (+ 487us) write_op.cc:270] Start()
0926 01:56:01.733634 (+ 102us) write_op.cc:276] Timestamp: P: 1790387742636802 usec, L: 0
0926 01:56:01.734133 (+ 499us) write_op.cc:270] Start()
0926 01:56:01.734216 (+ 83us) write_op.cc:276] Timestamp: P: 1790387742668208 usec, L: 0
0926 01:56:01.878033 (+143817us) write_op.cc:270] Start()
0926 01:56:01.878159 (+ 126us) write_op.cc:276] Timestamp: P: 1790387742717871 usec, L: 0
0926 01:56:01.884736 (+ 6577us) write_op.cc:270] Start()
0926 01:56:01.884829 (+ 93us) write_op.cc:276] Timestamp: P: 1790387742806392 usec, L: 0
0926 01:56:01.885424 (+ 595us) write_op.cc:270] Start()
0926 01:56:01.885515 (+ 91us) write_op.cc:276] Timestamp: P: 1790387742826811 usec, L: 0
0926 01:56:01.886043 (+ 528us) write_op.cc:270] Start()
0926 01:56:01.886146 (+ 103us) write_op.cc:276] Timestamp: P: 1790387742829110 usec, L: 0
0926 01:56:01.923733 (+ 37587us) write_op.cc:270] Start()
0926 01:56:01.923859 (+ 126us) write_op.cc:276] Timestamp: P: 1790387742831020 usec, L: 0
0926 01:56:01.924474 (+ 615us) write_op.cc:270] Start()
0926 01:56:01.924561 (+ 87us) write_op.cc:276] Timestamp: P: 1790387742947265 usec, L: 0
0926 01:56:01.925080 (+ 519us) write_op.cc:270] Start()
0926 01:56:01.925174 (+ 94us) write_op.cc:276] Timestamp: P: 1790387742979935 usec, L: 0
0926 01:56:01.925671 (+ 497us) write_op.cc:270] Start()
0926 01:56:01.925745 (+ 74us) write_op.cc:276] Timestamp: P: 1790387742982134 usec, L: 0
0926 01:56:01.926346 (+ 601us) write_op.cc:270] Start()
0926 01:56:01.961852 (+ 35506us) write_op.cc:276] Timestamp: P: 1790387743020043 usec, L: 0
0926 01:56:01.963508 (+ 1656us) write_op.cc:270] Start()
0926 01:56:01.963608 (+ 100us) write_op.cc:276] Timestamp: P: 1790387743054290 usec, L: 0
0926 01:56:01.964177 (+ 569us) write_op.cc:270] Start()
0926 01:56:01.964261 (+ 84us) write_op.cc:276] Timestamp: P: 1790387743083571 usec, L: 0
0926 01:56:01.964878 (+ 617us) write_op.cc:270] Start()
0926 01:56:01.964964 (+ 86us) write_op.cc:276] Timestamp: P: 1790387743183686 usec, L: 0
0926 01:56:01.966483 (+ 1519us) write_op.cc:270] Start()
0926 01:56:01.966571 (+ 88us) write_op.cc:276] Timestamp: P: 1790387743187007 usec, L: 0
0926 01:56:01.967073 (+ 502us) write_op.cc:270] Start()
0926 01:56:01.967176 (+ 103us) write_op.cc:276] Timestamp: P: 1790387743219209 usec, L: 0
0926 01:56:02.104362 (+137186us) write_op.cc:270] Start()
0926 01:56:02.104532 (+ 170us) write_op.cc:276] Timestamp: P: 1790387743336445 usec, L: 0
0926 01:56:02.105162 (+ 630us) write_op.cc:270] Start()
0926 01:56:02.105244 (+ 82us) write_op.cc:276] Timestamp: P: 1790387743396066 usec, L: 0
0926 01:56:02.105689 (+ 445us) write_op.cc:270] Start()
0926 01:56:02.105768 (+ 79us) write_op.cc:276] Timestamp: P: 1790387743459120 usec, L: 0
0926 01:56:02.106228 (+ 460us) write_op.cc:270] Start()
0926 01:56:02.106298 (+ 70us) write_op.cc:276] Timestamp: P: 1790387743507186 usec, L: 0
0926 01:56:02.159010 (+ 52712us) write_op.cc:270] Start()
0926 01:56:02.159125 (+ 115us) write_op.cc:276] Timestamp: P: 1790387743513829 usec, L: 0
0926 01:56:02.160588 (+ 1463us) write_op.cc:270] Start()
0926 01:56:02.160685 (+ 97us) write_op.cc:276] Timestamp: P: 1790387743521021 usec, L: 0
0926 01:56:02.161200 (+ 515us) write_op.cc:270] Start()
0926 01:56:02.161283 (+ 83us) write_op.cc:276] Timestamp: P: 1790387743523554 usec, L: 0
0926 01:56:02.184680 (+ 23397us) write_op.cc:270] Start()
0926 01:56:02.184783 (+ 103us) write_op.cc:276] Timestamp: P: 1790387743525400 usec, L: 0
0926 01:56:02.185408 (+ 625us) write_op.cc:270] Start()
0926 01:56:02.185507 (+ 99us) write_op.cc:276] Timestamp: P: 1790387743527242 usec, L: 0
0926 01:56:02.186077 (+ 570us) write_op.cc:270] Start()
0926 01:56:02.186174 (+ 97us) write_op.cc:276] Timestamp: P: 1790387743529196 usec, L: 0
0926 01:56:02.186640 (+ 466us) write_op.cc:270] Start()
0926 01:56:02.186721 (+ 81us) write_op.cc:276] Timestamp: P: 1790387743549228 usec, L: 0
0926 01:56:02.217544 (+ 30823us) write_op.cc:270] Start()
0926 01:56:02.217662 (+ 118us) write_op.cc:276] Timestamp: P: 1790387743672238 usec, L: 0
0926 01:56:02.231172 (+ 13510us) write_op.cc:270] Start()
0926 01:56:02.231282 (+ 110us) write_op.cc:276] Timestamp: P: 1790387743783259 usec, L: 0
0926 01:56:02.231857 (+ 575us) write_op.cc:270] Start()
0926 01:56:02.231935 (+ 78us) write_op.cc:276] Timestamp: P: 1790387743789588 usec, L: 0
0926 01:56:02.232419 (+ 484us) write_op.cc:270] Start()
0926 01:56:02.232495 (+ 76us) write_op.cc:276] Timestamp: P: 1790387743791702 usec, L: 0
0926 01:56:02.240782 (+ 8287us) write_op.cc:270] Start()
0926 01:56:02.240882 (+ 100us) write_op.cc:276] Timestamp: P: 1790387743801036 usec, L: 0
0926 01:56:02.241415 (+ 533us) write_op.cc:270] Start()
0926 01:56:02.241492 (+ 77us) write_op.cc:276] Timestamp: P: 1790387743893261 usec, L: 0
0926 01:56:02.242899 (+ 1407us) write_op.cc:270] Start()
0926 01:56:02.242982 (+ 83us) write_op.cc:276] Timestamp: P: 1790387744022756 usec, L: 0
0926 01:56:02.243457 (+ 475us) write_op.cc:270] Start()
0926 01:56:02.243535 (+ 78us) write_op.cc:276] Timestamp: P: 1790387744056695 usec, L: 0
0926 01:56:02.243975 (+ 440us) write_op.cc:270] Start()
0926 01:56:02.244048 (+ 73us) write_op.cc:276] Timestamp: P: 1790387744189603 usec, L: 0
0926 01:56:02.244475 (+ 427us) write_op.cc:270] Start()
0926 01:56:02.244545 (+ 70us) write_op.cc:276] Timestamp: P: 1790387744221174 usec, L: 0
0926 01:56:02.244965 (+ 420us) write_op.cc:270] Start()
0926 01:56:02.245042 (+ 77us) write_op.cc:276] Timestamp: P: 1790387744242107 usec, L: 0
0926 01:56:02.245473 (+ 431us) write_op.cc:270] Start()
0926 01:56:02.245545 (+ 72us) write_op.cc:276] Timestamp: P: 1790387744245813 usec, L: 0
0926 01:56:02.245957 (+ 412us) write_op.cc:270] Start()
0926 01:56:02.246024 (+ 67us) write_op.cc:276] Timestamp: P: 1790387744256605 usec, L: 0
0926 01:56:02.283940 (+ 37916us) write_op.cc:270] Start()
0926 01:56:02.284321 (+ 381us) write_op.cc:276] Timestamp: P: 1790387744261537 usec, L: 0
0926 01:56:02.285026 (+ 705us) write_op.cc:270] Start()
0926 01:56:02.285225 (+ 199us) write_op.cc:276] Timestamp: P: 1790387744307865 usec, L: 0
0926 01:56:02.295348 (+ 10123us) write_op.cc:270] Start()
0926 01:56:02.295456 (+ 108us) write_op.cc:276] Timestamp: P: 1790387744361051 usec, L: 0
0926 01:56:02.296080 (+ 624us) write_op.cc:270] Start()
0926 01:56:02.296177 (+ 97us) write_op.cc:276] Timestamp: P: 1790387744854480 usec, L: 0
0926 01:56:02.296679 (+ 502us) write_op.cc:270] Start()
0926 01:56:02.296767 (+ 88us) write_op.cc:276] Timestamp: P: 1790387744892007 usec, L: 0
0926 01:56:02.322714 (+ 25947us) write_op.cc:270] Start()
0926 01:56:02.322819 (+ 105us) write_op.cc:276] Timestamp: P: 1790387744916601 usec, L: 0
0926 01:56:02.323522 (+ 703us) write_op.cc:270] Start()
0926 01:56:02.323721 (+ 199us) write_op.cc:276] Timestamp: P: 1790387744919953 usec, L: 0
0926 01:56:02.324364 (+ 643us) write_op.cc:270] Start()
0926 01:56:02.324442 (+ 78us) write_op.cc:276] Timestamp: P: 1790387745215170 usec, L: 0
0926 01:56:02.331290 (+ 6848us) write_op.cc:270] Start()
0926 01:56:02.331378 (+ 88us) write_op.cc:276] Timestamp: P: 1790387745227242 usec, L: 0
0926 01:56:02.331885 (+ 507us) write_op.cc:270] Start()
0926 01:56:02.331959 (+ 74us) write_op.cc:276] Timestamp: P: 1790387745231039 usec, L: 0
0926 01:56:02.332404 (+ 445us) write_op.cc:270] Start()
0926 01:56:02.332468 (+ 64us) write_op.cc:276] Timestamp: P: 1790387745282715 usec, L: 0
0926 01:56:02.332876 (+ 408us) write_op.cc:270] Start()
0926 01:56:02.332956 (+ 80us) write_op.cc:276] Timestamp: P: 1790387745295692 usec, L: 0
0926 01:56:02.335858 (+ 2902us) write_op.cc:270] Start()
0926 01:56:02.335929 (+ 71us) write_op.cc:276] Timestamp: P: 1790387745400557 usec, L: 0
0926 01:56:02.336410 (+ 481us) write_op.cc:270] Start()
0926 01:56:02.336499 (+ 89us) write_op.cc:276] Timestamp: P: 1790387745448685 usec, L: 0
0926 01:56:02.336920 (+ 421us) write_op.cc:270] Start()
0926 01:56:02.336985 (+ 65us) write_op.cc:276] Timestamp: P: 1790387745487654 usec, L: 0
0926 01:56:02.337391 (+ 406us) write_op.cc:270] Start()
0926 01:56:02.337453 (+ 62us) write_op.cc:276] Timestamp: P: 1790387745523665 usec, L: 0
0926 01:56:02.337843 (+ 390us) write_op.cc:270] Start()
0926 01:56:02.337911 (+ 68us) write_op.cc:276] Timestamp: P: 1790387745527231 usec, L: 0
0926 01:56:02.338318 (+ 407us) write_op.cc:270] Start()
0926 01:56:02.338388 (+ 70us) write_op.cc:276] Timestamp: P: 1790387745618952 usec, L: 0
0926 01:56:02.350797 (+ 12409us) write_op.cc:270] Start()
0926 01:56:02.350893 (+ 96us) write_op.cc:276] Timestamp: P: 1790387745650854 usec, L: 0
0926 01:56:02.351669 (+ 776us) write_op.cc:270] Start()
0926 01:56:02.357318 (+ 5649us) write_op.cc:276] Timestamp: P: 1790387745675831 usec, L: 0
0926 01:56:02.358164 (+ 846us) write_op.cc:270] Start()
0926 01:56:02.358247 (+ 83us) write_op.cc:276] Timestamp: P: 1790387745734198 usec, L: 0
0926 01:56:02.358758 (+ 511us) write_op.cc:270] Start()
0926 01:56:02.358829 (+ 71us) write_op.cc:276] Timestamp: P: 1790387745955134 usec, L: 0
0926 01:56:02.359266 (+ 437us) write_op.cc:270] Start()
0926 01:56:02.359345 (+ 79us) write_op.cc:276] Timestamp: P: 1790387745961800 usec, L: 0
0926 01:56:02.367494 (+ 8149us) write_op.cc:270] Start()
0926 01:56:02.367595 (+ 101us) write_op.cc:276] Timestamp: P: 1790387746027700 usec, L: 0
0926 01:56:02.368154 (+ 559us) write_op.cc:270] Start()
0926 01:56:02.368222 (+ 68us) write_op.cc:276] Timestamp: P: 1790387746029975 usec, L: 0
0926 01:56:02.368673 (+ 451us) write_op.cc:270] Start()
0926 01:56:02.368742 (+ 69us) write_op.cc:276] Timestamp: P: 1790387746032519 usec, L: 0
0926 01:56:02.369182 (+ 440us) write_op.cc:270] Start()
0926 01:56:02.369250 (+ 68us) write_op.cc:276] Timestamp: P: 1790387746035944 usec, L: 0
0926 01:56:02.369643 (+ 393us) write_op.cc:270] Start()
0926 01:56:02.369706 (+ 63us) write_op.cc:276] Timestamp: P: 1790387746284931 usec, L: 0
0926 01:56:02.370096 (+ 390us) write_op.cc:270] Start()
0926 01:56:02.370179 (+ 83us) write_op.cc:276] Timestamp: P: 1790387746328353 usec, L: 0
0926 01:56:02.381161 (+ 10982us) write_op.cc:270] Start()
0926 01:56:02.381262 (+ 101us) write_op.cc:276] Timestamp: P: 1790387746351235 usec, L: 0
0926 01:56:02.381835 (+ 573us) write_op.cc:270] Start()
0926 01:56:02.381920 (+ 85us) write_op.cc:276] Timestamp: P: 1790387746386028 usec, L: 0
0926 01:56:02.382384 (+ 464us) write_op.cc:270] Start()
0926 01:56:02.382470 (+ 86us) write_op.cc:276] Timestamp: P: 1790387746565906 usec, L: 0
0926 01:56:02.382888 (+ 418us) write_op.cc:270] Start()
0926 01:56:02.382962 (+ 74us) write_op.cc:276] Timestamp: P: 1790387746585106 usec, L: 0
0926 01:56:02.383407 (+ 445us) write_op.cc:270] Start()
0926 01:56:02.383479 (+ 72us) write_op.cc:276] Timestamp: P: 1790387746616671 usec, L: 0
0926 01:56:02.383984 (+ 505us) write_op.cc:270] Start()
0926 01:56:02.384054 (+ 70us) write_op.cc:276] Timestamp: P: 1790387746641053 usec, L: 0
0926 01:56:02.384524 (+ 470us) write_op.cc:270] Start()
0926 01:56:02.384592 (+ 68us) write_op.cc:276] Timestamp: P: 1790387746688658 usec, L: 0
0926 01:56:02.399707 (+ 15115us) write_op.cc:270] Start()
0926 01:56:02.399807 (+ 100us) write_op.cc:276] Timestamp: P: 1790387746723919 usec, L: 0
0926 01:56:02.400641 (+ 834us) write_op.cc:270] Start()
0926 01:56:02.400724 (+ 83us) write_op.cc:276] Timestamp: P: 1790387746805117 usec, L: 0
0926 01:56:02.401196 (+ 472us) write_op.cc:270] Start()
0926 01:56:02.401266 (+ 70us) write_op.cc:276] Timestamp: P: 1790387746808177 usec, L: 0
0926 01:56:02.401700 (+ 434us) write_op.cc:270] Start()
0926 01:56:02.401767 (+ 67us) write_op.cc:276] Timestamp: P: 1790387746855356 usec, L: 0
0926 01:56:02.402225 (+ 458us) write_op.cc:270] Start()
0926 01:56:02.402302 (+ 77us) write_op.cc:276] Timestamp: P: 1790387746869324 usec, L: 0
0926 01:56:02.413228 (+ 10926us) write_op.cc:270] Start()
0926 01:56:02.413335 (+ 107us) write_op.cc:276] Timestamp: P: 1790387747031390 usec, L: 0
0926 01:56:02.413852 (+ 517us) write_op.cc:270] Start()
0926 01:56:02.413927 (+ 75us) write_op.cc:276] Timestamp: P: 1790387747060418 usec, L: 0
0926 01:56:02.414364 (+ 437us) write_op.cc:270] Start()
0926 01:56:02.414431 (+ 67us) write_op.cc:276] Timestamp: P: 1790387747136096 usec, L: 0
0926 01:56:02.414865 (+ 434us) write_op.cc:270] Start()
0926 01:56:02.414935 (+ 70us) write_op.cc:276] Timestamp: P: 1790387747151719 usec, L: 0
0926 01:56:02.415385 (+ 450us) write_op.cc:270] Start()
0926 01:56:02.415470 (+ 85us) write_op.cc:276] Timestamp: P: 1790387747235084 usec, L: 0
0926 01:56:02.415880 (+ 410us) write_op.cc:270] Start()
0926 01:56:02.415949 (+ 69us) write_op.cc:276] Timestamp: P: 1790387747275610 usec, L: 0
0926 01:56:02.416403 (+ 454us) write_op.cc:270] Start()
0926 01:56:02.416472 (+ 69us) write_op.cc:276] Timestamp: P: 1790387747375425 usec, L: 0
0926 01:56:02.416870 (+ 398us) write_op.cc:270] Start()
0926 01:56:02.416938 (+ 68us) write_op.cc:276] Timestamp: P: 1790387747431115 usec, L: 0
0926 01:56:02.417338 (+ 400us) write_op.cc:270] Start()
0926 01:56:02.417402 (+ 64us) write_op.cc:276] Timestamp: P: 1790387747465344 usec, L: 0
0926 01:56:02.444676 (+ 27274us) write_op.cc:270] Start()
0926 01:56:02.444803 (+ 127us) write_op.cc:276] Timestamp: P: 1790387747501924 usec, L: 0
0926 01:56:02.445477 (+ 674us) write_op.cc:270] Start()
0926 01:56:02.445553 (+ 76us) write_op.cc:276] Timestamp: P: 1790387747550881 usec, L: 0
0926 01:56:02.446011 (+ 458us) write_op.cc:270] Start()
0926 01:56:02.446083 (+ 72us) write_op.cc:276] Timestamp: P: 1790387747564859 usec, L: 0
0926 01:56:02.446528 (+ 445us) write_op.cc:270] Start()
0926 01:56:02.446592 (+ 64us) write_op.cc:276] Timestamp: P: 1790387747566828 usec, L: 0
0926 01:56:02.447001 (+ 409us) write_op.cc:270] Start()
0926 01:56:02.447067 (+ 66us) write_op.cc:276] Timestamp: P: 1790387747579896 usec, L: 0
0926 01:56:02.472122 (+ 25055us) write_op.cc:270] Start()
0926 01:56:02.472219 (+ 97us) write_op.cc:276] Timestamp: P: 1790387747581924 usec, L: 0
0926 01:56:02.472766 (+ 547us) write_op.cc:270] Start()
0926 01:56:02.472840 (+ 74us) write_op.cc:276] Timestamp: P: 1790387747609816 usec, L: 0
0926 01:56:02.473298 (+ 458us) write_op.cc:270] Start()
0926 01:56:02.473366 (+ 68us) write_op.cc:276] Timestamp: P: 1790387747641989 usec, L: 0
0926 01:56:02.473809 (+ 443us) write_op.cc:270] Start()
0926 01:56:02.473875 (+ 66us) write_op.cc:276] Timestamp: P: 1790387747753067 usec, L: 0
0926 01:56:02.474319 (+ 444us) write_op.cc:270] Start()
0926 01:56:02.474389 (+ 70us) write_op.cc:276] Timestamp: P: 1790387747763531 usec, L: 0
0926 01:56:02.474843 (+ 454us) write_op.cc:270] Start()
0926 01:56:02.474913 (+ 70us) write_op.cc:276] Timestamp: P: 1790387747768844 usec, L: 0
0926 01:56:02.475371 (+ 458us) write_op.cc:270] Start()
0926 01:56:02.488168 (+ 12797us) write_op.cc:276] Timestamp: P: 1790387747864322 usec, L: 0
0926 01:56:02.492637 (+ 4469us) write_op.cc:270] Start()
0926 01:56:02.492729 (+ 92us) write_op.cc:276] Timestamp: P: 1790387747887277 usec, L: 0
0926 01:56:02.493234 (+ 505us) write_op.cc:270] Start()
0926 01:56:02.493302 (+ 68us) write_op.cc:276] Timestamp: P: 1790387747895391 usec, L: 0
0926 01:56:02.493714 (+ 412us) write_op.cc:270] Start()
0926 01:56:02.493780 (+ 66us) write_op.cc:276] Timestamp: P: 1790387748000677 usec, L: 0
0926 01:56:02.495186 (+ 1406us) write_op.cc:270] Start()
0926 01:56:02.495256 (+ 70us) write_op.cc:276] Timestamp: P: 1790387748030156 usec, L: 0
0926 01:56:02.495681 (+ 425us) write_op.cc:270] Start()
0926 01:56:02.495748 (+ 67us) write_op.cc:276] Timestamp: P: 1790387748062049 usec, L: 0
0926 01:56:02.496192 (+ 444us) write_op.cc:270] Start()
0926 01:56:02.496256 (+ 64us) write_op.cc:276] Timestamp: P: 1790387748146297 usec, L: 0
0926 01:56:02.496638 (+ 382us) write_op.cc:270] Start()
0926 01:56:02.496709 (+ 71us) write_op.cc:276] Timestamp: P: 1790387748175815 usec, L: 0
0926 01:56:02.497134 (+ 425us) write_op.cc:270] Start()
0926 01:56:02.497197 (+ 63us) write_op.cc:276] Timestamp: P: 1790387748187033 usec, L: 0
0926 01:56:02.497577 (+ 380us) write_op.cc:270] Start()
0926 01:56:02.497646 (+ 69us) write_op.cc:276] Timestamp: P: 1790387748188993 usec, L: 0
0926 01:56:02.498028 (+ 382us) write_op.cc:270] Start()
0926 01:56:02.498090 (+ 62us) write_op.cc:276] Timestamp: P: 1790387748247810 usec, L: 0
0926 01:56:02.533934 (+ 35844us) write_op.cc:270] Start()
0926 01:56:02.534164 (+ 230us) write_op.cc:276] Timestamp: P: 1790387748365474 usec, L: 0
0926 01:56:02.534786 (+ 622us) write_op.cc:270] Start()
0926 01:56:02.534860 (+ 74us) write_op.cc:276] Timestamp: P: 1790387748398317 usec, L: 0
0926 01:56:02.535324 (+ 464us) write_op.cc:270] Start()
0926 01:56:02.535400 (+ 76us) write_op.cc:276] Timestamp: P: 1790387748574969 usec, L: 0
0926 01:56:02.535980 (+ 580us) write_op.cc:270] Start()
0926 01:56:02.536070 (+ 90us) write_op.cc:276] Timestamp: P: 1790387748578396 usec, L: 0
0926 01:56:02.536474 (+ 404us) write_op.cc:270] Start()
0926 01:56:02.536539 (+ 65us) write_op.cc:276] Timestamp: P: 1790387748686203 usec, L: 0
0926 01:56:02.536978 (+ 439us) write_op.cc:270] Start()
0926 01:56:02.537060 (+ 82us) write_op.cc:276] Timestamp: P: 1790387748834018 usec, L: 0
0926 01:56:02.537576 (+ 516us) write_op.cc:270] Start()
0926 01:56:02.537644 (+ 68us) write_op.cc:276] Timestamp: P: 1790387748838270 usec, L: 0
0926 01:56:02.538028 (+ 384us) write_op.cc:270] Start()
0926 01:56:02.538104 (+ 76us) write_op.cc:276] Timestamp: P: 1790387748884085 usec, L: 0
0926 01:56:02.538522 (+ 418us) write_op.cc:270] Start()
0926 01:56:02.538589 (+ 67us) write_op.cc:276] Timestamp: P: 1790387748985946 usec, L: 0
0926 01:56:02.538992 (+ 403us) write_op.cc:270] Start()
0926 01:56:02.539074 (+ 82us) write_op.cc:276] Timestamp: P: 1790387748998477 usec, L: 0
0926 01:56:02.539482 (+ 408us) write_op.cc:270] Start()
0926 01:56:02.539561 (+ 79us) write_op.cc:276] Timestamp: P: 1790387749000658 usec, L: 0
0926 01:56:02.539996 (+ 435us) write_op.cc:270] Start()
0926 01:56:02.540070 (+ 74us) write_op.cc:276] Timestamp: P: 1790387749042112 usec, L: 0
0926 01:56:02.540458 (+ 388us) write_op.cc:270] Start()
0926 01:56:02.540528 (+ 70us) write_op.cc:276] Timestamp: P: 1790387749220325 usec, L: 0
0926 01:56:02.581455 (+ 40927us) write_op.cc:270] Start()
0926 01:56:02.581565 (+ 110us) write_op.cc:276] Timestamp: P: 1790387749306055 usec, L: 0
0926 01:56:02.582179 (+ 614us) write_op.cc:270] Start()
0926 01:56:02.582262 (+ 83us) write_op.cc:276] Timestamp: P: 1790387749440041 usec, L: 0
0926 01:56:02.582755 (+ 493us) write_op.cc:270] Start()
0926 01:56:02.582823 (+ 68us) write_op.cc:276] Timestamp: P: 1790387749520361 usec, L: 0
0926 01:56:02.583283 (+ 460us) write_op.cc:270] Start()
0926 01:56:02.583361 (+ 78us) write_op.cc:276] Timestamp: P: 1790387749623862 usec, L: 0
0926 01:56:02.583808 (+ 447us) write_op.cc:270] Start()
0926 01:56:02.583875 (+ 67us) write_op.cc:276] Timestamp: P: 1790387749748689 usec, L: 0
0926 01:56:02.584305 (+ 430us) write_op.cc:270] Start()
0926 01:56:02.584378 (+ 73us) write_op.cc:276] Timestamp: P: 1790387749802482 usec, L: 0
0926 01:56:02.584782 (+ 404us) write_op.cc:270] Start()
0926 01:56:02.584848 (+ 66us) write_op.cc:276] Timestamp: P: 1790387749833055 usec, L: 0
0926 01:56:02.585284 (+ 436us) write_op.cc:270] Start()
0926 01:56:02.585358 (+ 74us) write_op.cc:276] Timestamp: P: 1790387749914243 usec, L: 0
0926 01:56:02.585779 (+ 421us) write_op.cc:270] Start()
0926 01:56:02.585846 (+ 67us) write_op.cc:276] Timestamp: P: 1790387750040441 usec, L: 0
0926 01:56:02.586270 (+ 424us) write_op.cc:270] Start()
0926 01:56:02.586343 (+ 73us) write_op.cc:276] Timestamp: P: 1790387750151755 usec, L: 0
0926 01:56:02.607006 (+ 20663us) write_op.cc:270] Start()
0926 01:56:02.607111 (+ 105us) write_op.cc:276] Timestamp: P: 1790387750300772 usec, L: 0
0926 01:56:02.607605 (+ 494us) write_op.cc:270] Start()
0926 01:56:02.607680 (+ 75us) write_op.cc:276] Timestamp: P: 1790387750329329 usec, L: 0
0926 01:56:02.608141 (+ 461us) write_op.cc:270] Start()
0926 01:56:02.608211 (+ 70us) write_op.cc:276] Timestamp: P: 1790387750465083 usec, L: 0
0926 01:56:02.608694 (+ 483us) write_op.cc:270] Start()
0926 01:56:02.608775 (+ 81us) write_op.cc:276] Timestamp: P: 1790387750546499 usec, L: 0
0926 01:56:02.609266 (+ 491us) write_op.cc:270] Start()
0926 01:56:02.609336 (+ 70us) write_op.cc:276] Timestamp: P: 1790387750623301 usec, L: 0
0926 01:56:02.609738 (+ 402us) write_op.cc:270] Start()
0926 01:56:02.609809 (+ 71us) write_op.cc:276] Timestamp: P: 1790387750699847 usec, L: 0
0926 01:56:02.610269 (+ 460us) write_op.cc:270] Start()
0926 01:56:02.610337 (+ 68us) write_op.cc:276] Timestamp: P: 1790387750815821 usec, L: 0
0926 01:56:02.613644 (+ 3307us) write_op.cc:270] Start()
0926 01:56:02.613719 (+ 75us) write_op.cc:276] Timestamp: P: 1790387750887912 usec, L: 0
0926 01:56:02.614151 (+ 432us) write_op.cc:270] Start()
0926 01:56:02.614225 (+ 74us) write_op.cc:
W20260926 01:56:06.724503 8843 consensus_peers.cc:597] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:06.734139 10343 consensus_queue.cc:1048] T 880d77d42c394e6a9cf0b3768a915402 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:06.762724 10127 raft_consensus.cc:493] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:06.763410 10127 raft_consensus.cc:515] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.766557 10127 leader_election.cc:290] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:06.766836 9340 tablet_bootstrap.cc:492] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:06.766795 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8e0e6ede1ce4487b08bfd5743e3656a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:06.767287 9340 ts_tablet_manager.cc:1403] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.000s
I20260926 01:56:06.767659 9212 raft_consensus.cc:2468] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:06.768853 9006 leader_election.cc:304] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:06.769380 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c8761a6c3f7435193a91bfc4fe7fa86" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:06.770015 8905 raft_consensus.cc:3060] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.769804 10127 raft_consensus.cc:2804] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:06.770809 10127 raft_consensus.cc:493] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:06.771139 10127 raft_consensus.cc:3060] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.772781 9341 tablet_bootstrap.cc:492] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:06.773231 9341 ts_tablet_manager.cc:1403] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.005s
I20260926 01:56:06.777387 8843 leader_election.cc:304] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:06.779842 10343 raft_consensus.cc:2804] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:06.780303 10343 raft_consensus.cc:493] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:06.780673 10343 raft_consensus.cc:3060] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.782707 9339 tablet_bootstrap.cc:492] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:06.783193 9339 ts_tablet_manager.cc:1403] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.000s
I20260926 01:56:06.789965 9339 raft_consensus.cc:359] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:06.790685 9339 raft_consensus.cc:385] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.790938 9339 raft_consensus.cc:740] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.794363 9339 consensus_queue.cc:260] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:06.796888 9339 ts_tablet_manager.cc:1434] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260926 01:56:06.797899 9339 tablet_bootstrap.cc:492] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:06.808804 8905 raft_consensus.cc:2468] T 1c8761a6c3f7435193a91bfc4fe7fa86 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:06.820060 9339 tablet_bootstrap.cc:654] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:06.821915 10327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 7 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:06.822906 10327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:06.827342 10358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:06.828094 10358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:06.857511 9341 raft_consensus.cc:359] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.870057 10246 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:06.870483 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]). Expected version 1 got 0
I20260926 01:56:06.870956 10246 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:06.870781 9341 raft_consensus.cc:385] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.871508 9341 raft_consensus.cc:740] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.871910 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:06.793918 9340 raft_consensus.cc:359] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:06.872502 9340 raft_consensus.cc:385] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:06.872785 9340 raft_consensus.cc:740] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:06.873154 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:06.874344 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:06.874173 9341 consensus_queue.cc:260] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.879164 9340 consensus_queue.cc:260] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:06.881739 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8e0e6ede1ce4487b08bfd5743e3656a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:06.882488 8904 raft_consensus.cc:2468] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:06.884089 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:06.886087 10343 raft_consensus.cc:515] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.893496 9341 ts_tablet_manager.cc:1434] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.120s user 0.003s sys 0.004s
I20260926 01:56:06.899448 9340 ts_tablet_manager.cc:1434] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.132s user 0.007s sys 0.000s
I20260926 01:56:06.900514 9340 tablet_bootstrap.cc:492] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:06.901466 10127 raft_consensus.cc:515] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.914029 10340 consensus_queue.cc:1048] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:06.906872 8537 catalog_manager.cc:5938] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:06.921252 9341 tablet_bootstrap.cc:492] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:06.935041 8539 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]). Expected version 1 got 0
I20260926 01:56:06.936085 8539 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]). Expected version 1 got 0
I20260926 01:56:06.936532 10343 leader_election.cc:290] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:06.936069 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d068e93fd374a6091d7473db069a7fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:06.937711 9218 raft_consensus.cc:3060] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:06.940887 8539 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]). Expected version 1 got 0
I20260926 01:56:06.946717 9218 raft_consensus.cc:2468] T 0d068e93fd374a6091d7473db069a7fa P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:06.948271 8843 leader_election.cc:304] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:06.949172 10343 raft_consensus.cc:2804] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:06.952576 10343 raft_consensus.cc:697] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:06.953482 10343 consensus_queue.cc:237] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:06.955607 9711 ts_tablet_manager.cc:1929] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:06.955999 9711 log.cc:1199] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/2e12827ad93c47cf9013e238215ceb7a
I20260926 01:56:06.956664 9711 ts_tablet_manager.cc:1950] T 2e12827ad93c47cf9013e238215ceb7a P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:06.971391 8539 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]). Expected version 1 got 0
I20260926 01:56:06.972287 8539 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 4623456312284a698ae69d4054909647 (table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa]). Expected version 1 got 0
I20260926 01:56:06.974253 8539 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:06.976758 10375 catalog_manager.cc:1471] Term changed from 6 to 7 while waiting for master leader catchup. Not loading sys catalog metadata
W20260926 01:56:06.990842 9003 leader_election.cc:343] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:06.991606 9003 leader_election.cc:304] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:06.992502 10355 raft_consensus.cc:2749] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:06.995667 9003 leader_election.cc:343] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: cbf8818e65d74a32ba21dcb8a9a56a62
I20260926 01:56:06.998617 9003 leader_election.cc:304] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:07.001579 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8e0e6ede1ce4487b08bfd5743e3656a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:07.002132 9218 raft_consensus.cc:3060] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:07.006479 9340 tablet_bootstrap.cc:654] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:07.009923 9218 raft_consensus.cc:2468] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:07.001446 10127 leader_election.cc:290] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:07.003234 9339 tablet_bootstrap.cc:492] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:07.019248 9339 ts_tablet_manager.cc:1403] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.222s user 0.014s sys 0.000s
I20260926 01:56:07.023432 9339 raft_consensus.cc:359] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.024241 9339 raft_consensus.cc:385] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:07.024564 9339 raft_consensus.cc:740] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:07.020217 10362 consensus_queue.cc:1048] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:07.035284 10355 raft_consensus.cc:2749] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:07.038285 9006 leader_election.cc:304] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:07.039407 10127 raft_consensus.cc:2804] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:07.039870 10127 raft_consensus.cc:697] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:07.040709 10127 consensus_queue.cc:237] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:07.047994 10340 consensus_queue.cc:1048] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:07.049510 10355 raft_consensus.cc:493] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:07.050060 10355 raft_consensus.cc:515] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.057231 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 201) took 9966 ms. Trace:
I20260926 01:56:07.057618 9711 rpcz_store.cc:276] 0926 01:55:57.090859 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.123792 (+ 32933us) service_pool.cc:224] Handling call
0926 01:56:06.682817 (+9559025us) ts_tablet_manager.cc:1151] Deleting tablet 2e12827ad93c47cf9013e238215ceb7a
0926 01:56:06.682885 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:06.788249 (+105364us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:06.955453 (+167204us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:07.057158 (+101705us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30134,"tablet-delete.queue_time_us":9558262}
I20260926 01:56:07.060738 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:07.062059 9003 leader_election.cc:343] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:07.062213 9006 leader_election.cc:343] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:07.062487 9003 leader_election.cc:304] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:07.063249 10355 leader_election.cc:290] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:07.067641 10404 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 7 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:07.068511 10404 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:07.086174 10362 consensus_queue.cc:1048] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260926 01:56:07.105325 9340 tablet_bootstrap.cc:492] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:07.105799 9340 ts_tablet_manager.cc:1403] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.205s user 0.008s sys 0.005s
I20260926 01:56:07.059501 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 2e12827ad93c47cf9013e238215ceb7a (table default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]) successfully deleted
W20260926 01:56:07.106775 8527 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 47306.9us
W20260926 01:56:07.084879 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 91964074a37441db8b09f78720159586 (table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 91964074a37441db8b09f78720159586 consensus state has no leader
I20260926 01:56:07.108054 10404 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:07.108352 9340 raft_consensus.cc:359] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.108964 9340 raft_consensus.cc:385] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:07.108666 10404 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:07.109232 9340 raft_consensus.cc:740] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:07.109527 8535 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 314) took 3424 ms. Trace:
I20260926 01:56:07.109871 9340 consensus_queue.cc:260] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.111912 8549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 354 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:07.115453 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 354 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:07.117098 9923 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:07.109846 8535 rpcz_store.cc:276] 0926 01:56:03.685441 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:05.504869 (+1819428us) service_pool.cc:224] Handling call
0926 01:56:05.506104 (+ 1235us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:56:05.511282 (+ 5178us) catalog_manager.cc:3764] Apply alter schema
0926 01:56:05.511887 (+ 605us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:56:05.512117 (+ 230us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:56:06.820534 (+1308417us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:56:07.109456 (+288922us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:05.539058 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:05.539585 (+ 527us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:05.539616 (+ 31us) write_op.cc:435] Acquired schema lock
0926 01:56:05.539630 (+ 14us) tablet.cc:662] Decoding operations
0926 01:56:05.539952 (+ 322us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:05.540000 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:05.540027 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:05.540205 (+ 178us) tablet.cc:701] Row locks acquired
0926 01:56:05.540222 (+ 17us) write_op.cc:260] PREPARE: finished
0926 01:56:05.540394 (+ 172us) write_op.cc:270] Start()
0926 01:56:05.540493 (+ 99us) write_op.cc:276] Timestamp: P: 1790387765540341 usec, L: 0
0926 01:56:05.540512 (+ 19us) op_driver.cc:348] REPLICATION: starting
0926 01:56:05.541103 (+ 591us) log.cc:844] Serialized 405 byte log entry
0926 01:56:06.798815 (+1257712us) op_driver.cc:464] REPLICATION: finished
0926 01:56:06.814637 (+ 15822us) write_op.cc:301] APPLY: starting
0926 01:56:06.814700 (+ 63us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:06.814792 (+ 92us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:06.815534 (+ 742us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:06.815783 (+ 249us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:06.815929 (+ 146us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:06.815971 (+ 42us) write_op.cc:312] APPLY: finished
0926 01:56:06.816186 (+ 215us) log.cc:844] Serialized 23 byte log entry
0926 01:56:06.818669 (+ 2483us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:06.818739 (+ 70us) write_op.cc:454] Released schema lock
0926 01:56:06.818774 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":15554,"apply.run_cpu_time_us":1574,"apply.run_wall_time_us":6256,"num_ops":1,"prepare.queue_time_us":25428,"prepare.run_cpu_time_us":2310,"prepare.run_wall_time_us":2310,"replication_time_us":1258183,"thread_start_us":17335,"threads_started":1}]]}
I20260926 01:56:07.025278 9339 consensus_queue.cc:260] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.127348 10404 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:07.127918 10404 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260926 01:56:07.129447 8540 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:42048 (request call id 308) took 4547 ms. Trace:
I20260926 01:56:07.129761 8540 rpcz_store.cc:276] 0926 01:56:02.582209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:02.878555 (+296346us) service_pool.cc:224] Handling call
0926 01:56:02.879624 (+ 1069us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:56:02.880324 (+ 700us) catalog_manager.cc:3764] Apply alter schema
0926 01:56:02.880984 (+ 660us) catalog_manager.cc:3796] Acquired catalog manager lock
0926 01:56:02.881168 (+ 184us) catalog_manager.cc:3954] Updating metadata on disk
0926 01:56:03.554082 (+672914us) catalog_manager.cc:3985] Committing alterations to in-memory state
0926 01:56:07.129248 (+3575166us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:03.008369 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:03.008811 (+ 442us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:03.008835 (+ 24us) write_op.cc:435] Acquired schema lock
0926 01:56:03.008848 (+ 13us) tablet.cc:662] Decoding operations
0926 01:56:03.009151 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:03.009197 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:03.009221 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:03.009402 (+ 181us) tablet.cc:701] Row locks acquired
0926 01:56:03.009418 (+ 16us) write_op.cc:260] PREPARE: finished
0926 01:56:03.009567 (+ 149us) write_op.cc:270] Start()
0926 01:56:03.009665 (+ 98us) write_op.cc:276] Timestamp: P: 1790387763009469 usec, L: 0
0926 01:56:03.009680 (+ 15us) op_driver.cc:348] REPLICATION: starting
0926 01:56:03.010268 (+ 588us) log.cc:844] Serialized 405 byte log entry
0926 01:56:03.549160 (+538892us) op_driver.cc:464] REPLICATION: finished
0926 01:56:03.550628 (+ 1468us) write_op.cc:301] APPLY: starting
0926 01:56:03.550761 (+ 133us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:03.550927 (+ 166us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:03.551126 (+ 199us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:03.551459 (+ 333us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:03.551729 (+ 270us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:03.551805 (+ 76us) write_op.cc:312] APPLY: finished
0926 01:56:03.552164 (+ 359us) log.cc:844] Serialized 23 byte log entry
0926 01:56:03.553448 (+ 1284us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:03.553596 (+ 148us) write_op.cc:454] Released schema lock
0926 01:56:03.553662 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:03.554520 (+ 858us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:03.554636 (+ 116us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1103,"apply.run_cpu_time_us":1794,"apply.run_wall_time_us":3844,"num_ops":1,"prepare.queue_time_us":125362,"prepare.run_cpu_time_us":2178,"prepare.run_wall_time_us":4189,"replication_time_us":539271}]]}
I20260926 01:56:07.158639 9340 ts_tablet_manager.cc:1434] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.053s user 0.008s sys 0.000s
I20260926 01:56:07.165575 9340 tablet_bootstrap.cc:492] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:07.170112 8538 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 46) took 5872 ms (client timeout 1000 ms). Trace:
I20260926 01:56:07.173624 8536 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 92896db04df346bf91f732b1ce89cd23
I20260926 01:56:07.173674 8537 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f] alter complete (version 3)
I20260926 01:56:07.176555 8536 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:42048 (request call id 311) took 4389 ms. Trace:
I20260926 01:56:07.180316 8874 tablet_service.cc:1558] Processing DeleteTablet for tablet 92896db04df346bf91f732b1ce89cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:03 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.183190 9340 tablet_bootstrap.cc:654] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:07.176875 8536 rpcz_store.cc:276] 0926 01:56:02.787639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.487712 (+700073us) service_pool.cc:224] Handling call
0926 01:56:03.535992 (+ 48280us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e607876 00007f717c1b6b34 00007f717c1b6ac2 00007f717c1b69f1 00007f717c1b696c 00007f717c1b5c22 00007f717a45c09c 00007f717a45b4d6
0926 01:56:03.537139 (+ 1147us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0926 01:56:03.537633 (+ 494us) catalog_manager.cc:2854] Modifying in-memory table state
0926 01:56:03.537761 (+ 128us) catalog_manager.cc:2863] Locking tablets
0926 01:56:04.042744 (+504983us) catalog_manager.cc:2877] Removing table and tablets from system table
0926 01:56:04.206640 (+163896us) catalog_manager.cc:2895] Removing table from by-name map
0926 01:56:04.206872 (+ 232us) catalog_manager.cc:2911] Committing in-memory state
0926 01:56:07.173359 (+2966487us) catalog_manager.cc:2915] Aborting table tasks
0926 01:56:07.176491 (+ 3132us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:04.188996 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:04.189510 (+ 514us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:04.189543 (+ 33us) write_op.cc:435] Acquired schema lock
0926 01:56:04.189557 (+ 14us) tablet.cc:662] Decoding operations
0926 01:56:04.189951 (+ 394us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:04.190010 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:04.190039 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:04.190406 (+ 367us) tablet.cc:701] Row locks acquired
0926 01:56:04.190427 (+ 21us) write_op.cc:260] PREPARE: finished
0926 01:56:04.190627 (+ 200us) write_op.cc:270] Start()
0926 01:56:04.190730 (+ 103us) write_op.cc:276] Timestamp: P: 1790387764190566 usec, L: 0
0926 01:56:04.190750 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:56:04.191357 (+ 607us) log.cc:844] Serialized 728 byte log entry
0926 01:56:04.202498 (+ 11141us) op_driver.cc:464] REPLICATION: finished
0926 01:56:04.202781 (+ 283us) write_op.cc:301] APPLY: starting
0926 01:56:04.202848 (+ 67us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:04.202967 (+ 119us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:04.203072 (+ 105us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:04.203369 (+ 297us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:04.203517 (+ 148us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0926 01:56:04.203565 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:56:04.203796 (+ 231us) log.cc:844] Serialized 29 byte log entry
0926 01:56:04.203958 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:04.204037 (+ 79us) write_op.cc:454] Released schema lock
0926 01:56:04.204080 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:04.311749 (+107669us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:04.311785 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":109372672,"child_traces":[["op",{"apply.queue_time_us":122,"apply.run_cpu_time_us":1515,"apply.run_wall_time_us":1515,"num_ops":2,"prepare.queue_time_us":144501,"prepare.run_cpu_time_us":2757,"prepare.run_wall_time_us":122700,"raft.queue_time_us":99,"raft.run_cpu_time_us":780,"raft.run_wall_time_us":780,"replication_time_us":11608}]]}
I20260926 01:56:07.192034 9187 tablet_service.cc:1558] Processing DeleteTablet for tablet 92896db04df346bf91f732b1ce89cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:03 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:07.208393 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet 1f6d9fedc7b241b59b25d591c7dc2fea with delete_type TABLET_DATA_DELETED (replaced by tablet af0840d6761a4521b0b6136b80eb6a6c) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.208711 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 3571967749844fd7859cf179a93cf403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:00 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.208745 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet ff004cf835c34095a836ad683cec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:58 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.211802 10395 raft_consensus.cc:2749] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:07.239158 10435 tablet_replica.cc:333] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:07.244285 10435 raft_consensus.cc:2243] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
W20260926 01:56:07.251099 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 189050us
I20260926 01:56:07.265913 10362 consensus_queue.cc:1048] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:07.270998 10432 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:07.271872 10432 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:07.273749 10435 raft_consensus.cc:2272] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:07.249485 9340 tablet_bootstrap.cc:492] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:07.276669 9340 ts_tablet_manager.cc:1403] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.000s
I20260926 01:56:07.278863 9341 tablet_bootstrap.cc:654] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:07.125388 9711 tablet_replica.cc:333] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:07.151546 9046 tablet_service.cc:1558] Processing DeleteTablet for tablet c36ea09cf15647c0a344c9ba9b289d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:48 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.152060 9043 tablet_service.cc:1558] Processing DeleteTablet for tablet c27f1dd7bb0449a6a3fa371a8636e09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.152343 9052 tablet_service.cc:1558] Processing DeleteTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 with delete_type TABLET_DATA_DELETED (replaced by tablet b778c95327fc477ca210cb3c983665c1) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.153543 9050 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ef5265eb6244aa96c8195aa30cdfb0 with delete_type TABLET_DATA_DELETED (replaced by tablet b612e8955d5f42dd8658320031fe36ca) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.164884 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bf2fa5fbe4146b5adb27ff09de878d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:50 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.165313 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 6752df6a58914df5b8718dd0439aa885) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.194490 9045 tablet_service.cc:1558] Processing DeleteTablet for tablet 92896db04df346bf91f732b1ce89cd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:03 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.291782 9340 raft_consensus.cc:359] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:07.292469 9340 raft_consensus.cc:385] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:07.292733 9340 raft_consensus.cc:740] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:07.293354 9340 consensus_queue.cc:260] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:07.313580 9340 ts_tablet_manager.cc:1434] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.036s user 0.004s sys 0.002s
I20260926 01:56:07.314674 9340 tablet_bootstrap.cc:492] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:07.324328 8524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 48237.4us
I20260926 01:56:07.327845 10397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:07.328642 10397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:07.337563 9218 raft_consensus.cc:3060] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:07.340410 9340 tablet_bootstrap.cc:654] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:07.346750 8537 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c] alter complete (version 1)
I20260926 01:56:07.346873 9339 ts_tablet_manager.cc:1434] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.327s user 0.007s sys 0.000s
I20260926 01:56:07.348043 9339 tablet_bootstrap.cc:492] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:07.349594 9711 raft_consensus.cc:2243] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:07.350257 9711 raft_consensus.cc:2272] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:07.362077 10421 raft_consensus.cc:493] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:07.362689 10421 raft_consensus.cc:515] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:07.364830 10421 leader_election.cc:290] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:07.368773 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:07.370774 8843 leader_election.cc:343] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:07.375212 8537 catalog_manager.cc:6391] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409] alter complete (version 1)
W20260926 01:56:07.431306 9155 consensus_peers.cc:597] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e12827ad93c47cf9013e238215ceb7a. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:07.433045 9711 ts_tablet_manager.cc:1916] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:07.171015 8538 rpcz_store.cc:269] 0926 01:56:01.297748 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.354440 (+ 56692us) service_pool.cc:224] Handling call
0926 01:56:07.169860 (+5815420us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:04.312273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:04.312811 (+ 538us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:04.312838 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:56:04.312851 (+ 13us) tablet.cc:662] Decoding operations
0926 01:56:04.313381 (+ 530us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:04.313441 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:04.313473 (+ 32us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:04.313850 (+ 377us) tablet.cc:701] Row locks acquired
0926 01:56:04.313868 (+ 18us) write_op.cc:260] PREPARE: finished
0926 01:56:04.314011 (+ 143us) write_op.cc:270] Start()
0926 01:56:04.314115 (+ 104us) write_op.cc:276] Timestamp: P: 1790387764313950 usec, L: 0
0926 01:56:04.314135 (+ 20us) op_driver.cc:348] REPLICATION: starting
0926 01:56:04.314779 (+ 644us) log.cc:844] Serialized 530 byte log entry
0926 01:56:04.488722 (+173943us) op_driver.cc:464] REPLICATION: finished
0926 01:56:04.489139 (+ 417us) write_op.cc:301] APPLY: starting
0926 01:56:04.489209 (+ 70us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:04.489327 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:04.489444 (+ 117us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:04.489660 (+ 216us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:04.489807 (+ 147us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:04.489846 (+ 39us) write_op.cc:312] APPLY: finished
0926 01:56:04.490076 (+ 230us) log.cc:844] Serialized 23 byte log entry
0926 01:56:04.490243 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:04.490318 (+ 75us) write_op.cc:454] Released schema lock
0926 01:56:04.490356 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:04.536476 (+ 46120us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:04.536511 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":227,"apply.run_cpu_time_us":1507,"apply.run_wall_time_us":1506,"num_ops":1,"prepare.queue_time_us":72165,"prepare.run_cpu_time_us":2643,"prepare.run_wall_time_us":224168,"raft.queue_time_us":162798,"raft.run_cpu_time_us":873,"raft.run_wall_time_us":874,"replication_time_us":174474}]]}
W20260926 01:56:07.453673 8531 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 47) took 6066 ms (client timeout 1000 ms). Trace:
I20260926 01:56:07.468954 10436 tablet_replica.cc:333] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:07.475665 8882 tablet_service.cc:1558] Processing DeleteTablet for tablet f100eb5dc9a8452e943d4cce5284bc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:52 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.476357 8879 tablet_service.cc:1558] Processing DeleteTablet for tablet 943a990fa10948e8961c02e9999569f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:51 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.479612 8881 tablet_service.cc:1558] Processing DeleteTablet for tablet 1743e863cdd84f3f8ca413fa81cd1914 with delete_type TABLET_DATA_DELETED (replaced by tablet b778c95327fc477ca210cb3c983665c1) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.461474 10339 raft_consensus.cc:493] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:07.480358 10339 raft_consensus.cc:515] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.470130 10436 raft_consensus.cc:2243] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:07.492707 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:07.492980 10436 raft_consensus.cc:2272] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 01:56:07.495198 8843 leader_election.cc:343] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:07.495749 8884 tablet_service.cc:1558] Processing DeleteTablet for tablet 17b8b0d54f5645e5963f1de217f94c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 6752df6a58914df5b8718dd0439aa885) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.495872 8874 tablet_service.cc:1558] Processing DeleteTablet for tablet 72f666b1c069435b8ff0a3fae9f1e873 with delete_type TABLET_DATA_DELETED (replaced by tablet e76071c66b1d4635ac1a28110eb684fa) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:07.499842 10426 consensus_queue.cc:1048] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:07.443172 10127 raft_consensus.cc:493] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:07.450384 9045 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f49a5a10ea4f7a821d89f18e3af560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:54 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.467183 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:07.469348 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d068e93fd374a6091d7473db069a7fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:07.486814 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d56f3a5dfed4ac78f1da92e22db558a with delete_type TABLET_DATA_DELETED (replaced by tablet 1c5c66728bb54ec6beb4f0a2c25d9105) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.511305 9051 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e12827ad93c47cf9013e238215ceb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:56 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:07.505782 10127 raft_consensus.cc:515] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.515695 10127 leader_election.cc:290] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:07.507074 9069 raft_consensus.cc:3060] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:07.545275 10455 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260926 01:56:07.629272 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:07.482348 10339 leader_election.cc:290] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:07.636207 9339 tablet_bootstrap.cc:654] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:07.641013 9340 tablet_bootstrap.cc:492] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:07.641618 9340 ts_tablet_manager.cc:1403] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.327s user 0.024s sys 0.036s
I20260926 01:56:07.646684 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8e0e6ede1ce4487b08bfd5743e3656a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:07.647241 8907 raft_consensus.cc:3060] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:07.646265 9006 leader_election.cc:343] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:07.644559 9340 raft_consensus.cc:359] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.650002 9340 raft_consensus.cc:385] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:07.650277 9340 raft_consensus.cc:740] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:07.656904 9340 consensus_queue.cc:260] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:07.457428 8537 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.129:35729 (request call id 49) took 3504 ms (client timeout 1000 ms). Trace:
W20260926 01:56:07.522212 8534 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 48) took 3857 ms (client timeout 1000 ms). Trace:
I20260926 01:56:07.664716 8907 raft_consensus.cc:2468] T e8e0e6ede1ce4487b08bfd5743e3656a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
W20260926 01:56:07.661783 8534 rpcz_store.cc:269] 0926 01:56:03.664770 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.903992 (+239222us) service_pool.cc:224] Handling call
0926 01:56:07.389269 (+3485277us) spinlock_profiling.cc:243] Waited 9.41 ms on lock 0x7b6800037160. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717e525b81 00007f717e501c8a 00007f717e5003c8 00007f717e4fea00 00007f717e6068de 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c
0926 01:56:07.522120 (+132851us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:06.479295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:06.479853 (+ 558us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:06.479881 (+ 28us) write_op.cc:435] Acquired schema lock
0926 01:56:06.479901 (+ 20us) tablet.cc:662] Decoding operations
0926 01:56:06.480509 (+ 608us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:06.480559 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:06.480586 (+ 27us) tablet.cc:685] Acquiring locks for 4 operations
0926 01:56:06.480995 (+ 409us) tablet.cc:701] Row locks acquired
0926 01:56:06.481014 (+ 19us) write_op.cc:260] PREPARE: finished
0926 01:56:06.481163 (+ 149us) write_op.cc:270] Start()
0926 01:56:06.481271 (+ 108us) write_op.cc:276] Timestamp: P: 1790387766481078 usec, L: 0
0926 01:56:06.481287 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:56:06.481946 (+ 659us) log.cc:844] Serialized 1662 byte log entry
0926 01:56:06.817577 (+335631us) op_driver.cc:464] REPLICATION: finished
0926 01:56:06.821028 (+ 3451us) write_op.cc:301] APPLY: starting
0926 01:56:06.821090 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:06.821271 (+ 181us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:06.821380 (+ 109us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:06.821901 (+ 521us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:06.822052 (+ 151us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0926 01:56:06.822094 (+ 42us) write_op.cc:312] APPLY: finished
0926 01:56:06.822332 (+ 238us) log.cc:844] Serialized 41 byte log entry
0926 01:56:06.822498 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:06.822604 (+ 106us) write_op.cc:454] Released schema lock
0926 01:56:06.822640 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":3975,"reported_tablets":35,"spinlock_wait_cycles":21635328,"child_traces":[["op",{"apply.queue_time_us":3293,"apply.run_cpu_time_us":1984,"apply.run_wall_time_us":73565,"num_ops":4,"prepare.queue_time_us":72835,"prepare.run_cpu_time_us":3314,"prepare.run_wall_time_us":64924,"raft.queue_time_us":4277,"raft.run_cpu_time_us":1209,"raft.run_wall_time_us":1209,"replication_time_us":336186,"thread_start_us":130004,"threads_started":2}]]}
I20260926 01:56:07.670010 10453 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20260926 01:56:07.661715 8537 rpcz_store.cc:269] 0926 01:56:03.953365 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:04.071798 (+118433us) service_pool.cc:224] Handling call
0926 01:56:04.184107 (+112309us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:07.457332 (+3273225us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:06.971215 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:06.971750 (+ 535us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:06.971778 (+ 28us) write_op.cc:435] Acquired schema lock
0926 01:56:06.971793 (+ 15us) tablet.cc:662] Decoding operations
0926 01:56:06.972114 (+ 321us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:06.972164 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:06.972189 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:06.972382 (+ 193us) tablet.cc:701] Row locks acquired
0926 01:56:06.972400 (+ 18us) write_op.cc:260] PREPARE: finished
0926 01:56:06.972545 (+ 145us) write_op.cc:270] Start()
0926 01:56:06.972635 (+ 90us) write_op.cc:276] Timestamp: P: 1790387766972460 usec, L: 0
0926 01:56:06.972653 (+ 18us) op_driver.cc:348] REPLICATION: starting
0926 01:56:06.973291 (+ 638us) log.cc:844] Serialized 356 byte log entry
0926 01:56:06.991280 (+ 17989us) op_driver.cc:464] REPLICATION: finished
0926 01:56:06.992540 (+ 1260us) write_op.cc:301] APPLY: starting
0926 01:56:06.992612 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:06.992714 (+ 102us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:06.992817 (+ 103us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:06.993028 (+ 211us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:06.993157 (+ 129us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:06.993210 (+ 53us) write_op.cc:312] APPLY: finished
0926 01:56:06.993404 (+ 194us) log.cc:844] Serialized 23 byte log entry
0926 01:56:06.993553 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:06.993622 (+ 69us) write_op.cc:454] Released schema lock
0926 01:56:06.993654 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:06.993898 (+ 244us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:06.993924 (+ 26us) write_op.cc:454] Released schema lock
Related trace 'op':
0926 01:56:07.190833 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:07.191311 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:07.191338 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:56:07.191350 (+ 12us) tablet.cc:662] Decoding operations
0926 01:56:07.191627 (+ 277us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:07.191672 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:07.191696 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:07.191856 (+ 160us) tablet.cc:701] Row locks acquired
0926 01:56:07.191877 (+ 21us) write_op.cc:260] PREPARE: finished
0926 01:56:07.192010 (+ 133us) write_op.cc:270] Start()
0926 01:56:07.192102 (+ 92us) write_op.cc:276] Timestamp: P: 1790387767191936 usec, L: 0
0926 01:56:07.192119 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:56:07.248763 (+ 56644us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b5400030d20. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c70aa50 00007f717c794302 00007f717c789727 00007f717c789564 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11
0926 01:56:07.262595 (+ 13832us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b5400030d20. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f6aa1 00007f717c7053f4 00007f717c789097 00007f717c7895c9 00007f717cc19791 00007f717cc18f68 00007f717cc1dc01 00007f717cc1dbb9 00007f717cc1db49 00007f717cc1db11 00007f717cc1cd3d
0926 01:56:07.263175 (+ 580us) log.cc:844] Serialized 356 byte log entry
0926 01:56:07.342593 (+ 79418us) op_driver.cc:464] REPLICATION: finished
0926 01:56:07.342923 (+ 330us) write_op.cc:301] APPLY: starting
0926 01:56:07.342978 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:07.343076 (+ 98us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:07.343095 (+ 19us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:07.343329 (+ 234us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:07.343455 (+ 126us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:07.343497 (+ 42us) write_op.cc:312] APPLY: finished
0926 01:56:07.343689 (+ 192us) log.cc:844] Serialized 23 byte log entry
0926 01:56:07.343856 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:07.343925 (+ 69us) write_op.cc:454] Released schema lock
0926 01:56:07.343956 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:07.344202 (+ 246us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:07.344226 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0926 01:56:07.359224 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:07.359734 (+ 510us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:07.359763 (+ 29us) write_op.cc:435] Acquired schema lock
0926 01:56:07.359779 (+ 16us) tablet.cc:662] Decoding operations
0926 01:56:07.360080 (+ 301us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:07.360127 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:07.360160 (+ 33us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:07.360341 (+ 181us) tablet.cc:701] Row locks acquired
0926 01:56:07.360353 (+ 12us) write_op.cc:260] PREPARE: finished
0926 01:56:07.360495 (+ 142us) write_op.cc:270] Start()
0926 01:56:07.360587 (+ 92us) write_op.cc:276] Timestamp: P: 1790387767360402 usec, L: 0
0926 01:56:07.360606 (+ 19us) op_driver.cc:348] REPLICATION: starting
0926 01:56:07.361195 (+ 589us) log.cc:844] Serialized 356 byte log entry
0926 01:56:07.370570 (+ 9375us) op_driver.cc:464] REPLICATION: finished
0926 01:56:07.370923 (+ 353us) write_op.cc:301] APPLY: starting
0926 01:56:07.370985 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:07.371094 (+ 109us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:07.371200 (+ 106us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:07.371396 (+ 196us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:07.371553 (+ 157us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:07.371597 (+ 44us) write_op.cc:312] APPLY: finished
0926 01:56:07.371785 (+ 188us) log.cc:844] Serialized 23 byte log entry
0926 01:56:07.371941 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:07.372013 (+ 72us) write_op.cc:454] Released schema lock
0926 01:56:07.372045 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:07.372334 (+ 289us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:07.372360 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3475,"reported_tablets":21,"spinlock_wait_cycles":249045888,"child_traces":[["op",{"apply.queue_time_us":1081,"apply.run_cpu_time_us":1327,"apply.run_wall_time_us":1327,"num_ops":1,"prepare.queue_time_us":274,"prepare.run_cpu_time_us":2452,"prepare.run_wall_time_us":2452,"raft.queue_time_us":1341,"raft.run_cpu_time_us":791,"raft.run_wall_time_us":790,"replication_time_us":18487}],["op",{"apply.queue_time_us":184,"apply.run_cpu_time_us":1236,"apply.run_wall_time_us":1236,"num_ops":1,"prepare.queue_time_us":5870,"prepare.run_cpu_time_us":14375,"prepare.run_wall_time_us":138346,"raft.queue_time_us":33871,"raft.run_cpu_time_us":827,"raft.run_wall_time_us":1210,"replication_time_us":150321,"spinlock_wait_cycles":157209728,"thread_start_us":65549,"threads_started":1}],["op",{"apply.queue_time_us":192,"apply.run_cpu_time_us":1378,"apply.run_wall_time_us":1377,"num_ops":1,"prepare.queue_time_us":9269,"prepare.run_cpu_time_us":2349,"prepare.run_wall_time_us":3412,"raft.queue_time_us":302,"raft.run_cpu_time_us":732,"raft.run_wall_time_us":783,"replication_time_us":9801}]]}
I20260926 01:56:07.647466 10339 raft_consensus.cc:493] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:07.673658 10339 raft_consensus.cc:515] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:07.674896 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 126015us
I20260926 01:56:07.675791 10339 leader_election.cc:290] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:07.687219 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:07.687997 8899 raft_consensus.cc:2468] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:07.691437 10340 raft_consensus.cc:493] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:07.691972 10340 raft_consensus.cc:515] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.648852 10435 ts_tablet_manager.cc:1916] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:07.694206 10340 leader_election.cc:290] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:07.700560 10339 consensus_queue.cc:1048] T dbf222f9e14d4332b679f7ee94af1e0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20260926 01:56:07.744544 10343 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.018s sys 0.033s
W20260926 01:56:07.744935 10343 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s user 0.018s sys 0.033s
I20260926 01:56:07.748769 10436 ts_tablet_manager.cc:1916] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:07.762136 10453 tablet.cc:1751] T a09351c3db384b4383ca5ff0eaca982c P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:56:07.788529 10436 ts_tablet_manager.cc:1929] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:07.789229 10436 log.cc:1199] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/92896db04df346bf91f732b1ce89cd23
I20260926 01:56:07.790278 10436 ts_tablet_manager.cc:1950] T 92896db04df346bf91f732b1ce89cd23 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:07.793761 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 92896db04df346bf91f732b1ce89cd23 (table default.Table_129e4100f7f245e9877924fd11902aea [id=6aedab17761246a7aace66b2cfd6d09e]) successfully deleted
I20260926 01:56:07.864955 9341 tablet_bootstrap.cc:492] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:07.865453 9341 ts_tablet_manager.cc:1403] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.944s user 0.010s sys 0.005s
W20260926 01:56:07.913395 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 264924us
I20260926 01:56:07.915459 10339 consensus_queue.cc:1048] T 0fdb7a7328c54031bd3afac1636d0f75 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:07.661680 8531 rpcz_store.cc:269] 0926 01:56:01.387373 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.674256 (+286883us) service_pool.cc:224] Handling call
0926 01:56:01.902708 (+228452us) spinlock_profiling.cc:243] Waited 70.3 ms on lock 0x7b5400030554. stack: 00007f71755f9323 00007f71755f9220 00007f717519d574 0000000000516964 00000000005168f9 00007f717c6f75c0 00007f717c7966fa 00007f717e507d8c 00007f717e60601e 00007f717c1aafb4 00007f717c1aaf42 00007f717c1aae71 00007f717c1aadec 00007f717c1aa0a2 00007f717a45c09c 00007f717a45b4d6
0926 01:56:07.453595 (+5550887us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:05.068297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:05.068780 (+ 483us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:05.068807 (+ 27us) write_op.cc:435] Acquired schema lock
0926 01:56:05.068826 (+ 19us) tablet.cc:662] Decoding operations
0926 01:56:05.069146 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:05.069189 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:05.069222 (+ 33us) tablet.cc:685] Acquiring locks for 1 operations
0926 01:56:05.069415 (+ 193us) tablet.cc:701] Row locks acquired
0926 01:56:05.069433 (+ 18us) write_op.cc:260] PREPARE: finished
0926 01:56:05.069593 (+ 160us) write_op.cc:270] Start()
0926 01:56:05.069826 (+ 233us) write_op.cc:276] Timestamp: P: 1790387765069459 usec, L: 0
0926 01:56:05.069842 (+ 16us) op_driver.cc:348] REPLICATION: starting
0926 01:56:05.070441 (+ 599us) log.cc:844] Serialized 530 byte log entry
0926 01:56:05.630325 (+559884us) op_driver.cc:464] REPLICATION: finished
0926 01:56:05.630653 (+ 328us) write_op.cc:301] APPLY: starting
0926 01:56:05.630739 (+ 86us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:05.630851 (+ 112us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:05.630972 (+ 121us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:05.631214 (+ 242us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:05.631367 (+ 153us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0926 01:56:05.631415 (+ 48us) write_op.cc:312] APPLY: finished
0926 01:56:05.631621 (+ 206us) log.cc:844] Serialized 23 byte log entry
0926 01:56:05.631780 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:05.631852 (+ 72us) write_op.cc:454] Released schema lock
0926 01:56:05.631885 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0926 01:56:05.632159 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:05.632188 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12663,"reported_tablets":22,"spinlock_wait_cycles":161687424,"child_traces":[["op",{"apply.queue_time_us":123,"apply.run_cpu_time_us":1473,"apply.run_wall_time_us":1474,"num_ops":1,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":2298,"prepare.run_wall_time_us":3558,"replication_time_us":560351}]]}
I20260926 01:56:07.918956 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:07.926208 9340 ts_tablet_manager.cc:1434] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.284s user 0.010s sys 0.001s
I20260926 01:56:07.944109 10463 tablet.cc:1751] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:56:07.951607 9340 tablet_bootstrap.cc:492] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:07.951843 9341 raft_consensus.cc:359] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:07.952489 9341 raft_consensus.cc:385] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:07.952739 9341 raft_consensus.cc:740] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:07.957753 9340 tablet_bootstrap.cc:654] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:07.972553 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:07.973141 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:07.973290 8899 raft_consensus.cc:2468] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:56:07.974833 8843 leader_election.cc:343] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:07.985883 9069 raft_consensus.cc:2468] T 0d068e93fd374a6091d7473db069a7fa P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:07.988664 10475 tablet.cc:1751] T 04e79a2e52fb40619b2bc99396abda5b P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260926 01:56:08.010763 9341 consensus_queue.cc:260] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.011734 9218 raft_consensus.cc:2468] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:08.016242 9341 ts_tablet_manager.cc:1434] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.151s user 0.006s sys 0.001s
W20260926 01:56:08.027521 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 52110.3us
W20260926 01:56:08.037338 8843 leader_election.cc:343] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:08.039121 9711 ts_tablet_manager.cc:1929] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 01:56:08.039635 9711 log.cc:1199] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/b659eca3c97144aaacd41e747aecd96d
I20260926 01:56:08.041186 8843 leader_election.cc:304] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:08.043830 9711 ts_tablet_manager.cc:1950] T b659eca3c97144aaacd41e747aecd96d P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:08.044443 10340 raft_consensus.cc:2749] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:08.055526 9341 tablet_bootstrap.cc:492] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:08.020869 8539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.76.130:57131 (request call id 47) took 5399 ms (client timeout 1000 ms). Trace:
W20260926 01:56:08.087260 8539 rpcz_store.cc:269] 0926 01:56:02.620997 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:02.621124 (+ 127us) service_pool.cc:224] Handling call
0926 01:56:08.020778 (+5399654us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":32}
I20260926 01:56:08.089005 10398 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
W20260926 01:56:08.125322 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 consensus state has no leader
I20260926 01:56:08.137466 10435 ts_tablet_manager.cc:1929] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:08.137925 10435 log.cc:1199] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/92896db04df346bf91f732b1ce89cd23
I20260926 01:56:08.142613 9341 tablet_bootstrap.cc:654] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:08.140167 8843 leader_election.cc:304] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:08.067739 10473 consensus_queue.cc:1048] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:08.147797 10343 raft_consensus.cc:2804] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:08.148150 10343 raft_consensus.cc:493] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:08.148444 10343 raft_consensus.cc:3060] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.063583 10355 raft_consensus.cc:493] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:08.149799 10355 raft_consensus.cc:515] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.151907 10355 leader_election.cc:290] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:08.062160 10395 raft_consensus.cc:493] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:08.152570 10395 raft_consensus.cc:515] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:08.154528 10395 leader_election.cc:290] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:08.075196 9339 tablet_bootstrap.cc:492] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:08.155213 9339 ts_tablet_manager.cc:1403] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.808s user 0.010s sys 0.002s
I20260926 01:56:08.155335 10435 ts_tablet_manager.cc:1950] T 92896db04df346bf91f732b1ce89cd23 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:08.157814 9339 raft_consensus.cc:359] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.158598 9339 raft_consensus.cc:385] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:08.158886 9339 raft_consensus.cc:740] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:08.159677 9339 consensus_queue.cc:260] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.161798 9339 ts_tablet_manager.cc:1434] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 01:56:08.108863 9003 leader_election.cc:304] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:08.117193 10127 consensus_queue.cc:1048] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:56:08.163296 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 54558.3us
I20260926 01:56:08.163690 10127 raft_consensus.cc:2804] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:08.164047 10127 raft_consensus.cc:493] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:08.164366 10127 raft_consensus.cc:3060] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.169831 9339 tablet_bootstrap.cc:492] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:08.175637 10127 raft_consensus.cc:515] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.176887 9003 leader_election.cc:304] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:08.177731 10395 raft_consensus.cc:2804] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:08.178059 10395 raft_consensus.cc:493] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:08.178357 10395 raft_consensus.cc:3060] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.184152 10340 consensus_queue.cc:1048] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:08.187743 9341 tablet_bootstrap.cc:492] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:08.188237 9341 ts_tablet_manager.cc:1403] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.005s
I20260926 01:56:08.158787 10343 raft_consensus.cc:515] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.147105 10463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51458 (request call id 262) took 1033 ms. Trace:
I20260926 01:56:08.194938 10343 leader_election.cc:290] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:08.194613 10463 rpcz_store.cc:276] 0926 01:56:07.113945 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.114102 (+ 157us) service_pool.cc:224] Handling call
0926 01:56:08.147021 (+1032919us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:07.146902 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:56:07.147260 (+ 358us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:56:07.147302 (+ 42us) alter_schema_op.cc:66] Acquired schema lock
0926 01:56:07.147347 (+ 45us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:56:07.147481 (+ 134us) alter_schema_op.cc:128] START. Timestamp: 1790387767147392
0926 01:56:07.147511 (+ 30us) op_driver.cc:348] REPLICATION: starting
0926 01:56:07.148146 (+ 635us) log.cc:844] Serialized 173 byte log entry
0926 01:56:07.650962 (+502816us) op_driver.cc:464] REPLICATION: finished
0926 01:56:07.943885 (+292923us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:56:07.945447 (+ 1562us) log.cc:844] Serialized 5 byte log entry
0926 01:56:08.104287 (+158840us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.105057 (+ 770us) log.cc:844] Serialized 14 byte log entry
0926 01:56:08.146843 (+ 41786us) alter_schema_op.cc:73] Released schema lock
0926 01:56:08.146881 (+ 38us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":292709,"mutex_wait_us":25739,"prepare.queue_time_us":31658,"prepare.run_cpu_time_us":2256,"prepare.run_wall_time_us":73705,"raft.queue_time_us":27663,"raft.run_cpu_time_us":2208,"raft.run_wall_time_us":2209,"replication_time_us":503303,"thread_start_us":382537,"threads_started":3,"wal-append.queue_time_us":10153}]]}
I20260926 01:56:08.202149 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:08.202484 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:08.202837 9069 raft_consensus.cc:2468] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:08.203265 9071 raft_consensus.cc:2393] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:08.206589 10127 leader_election.cc:290] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:08.207684 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 204) took 10989 ms. Trace:
I20260926 01:56:08.208813 10435 tablet_replica.cc:333] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:08.215260 10435 raft_consensus.cc:2243] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:08.217998 9711 rpcz_store.cc:276] 0926 01:55:57.218281 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:57.225365 (+ 7084us) service_pool.cc:224] Handling call
0926 01:56:07.125190 (+9899825us) ts_tablet_manager.cc:1151] Deleting tablet b659eca3c97144aaacd41e747aecd96d
0926 01:56:07.125259 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:07.946826 (+821567us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.038961 (+ 92135us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.207607 (+168646us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64594,"tablet-delete.queue_time_us":9899212}
I20260926 01:56:08.217621 9340 tablet_bootstrap.cc:492] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:08.219825 9340 ts_tablet_manager.cc:1403] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.268s user 0.011s sys 0.000s
I20260926 01:56:08.220378 10435 raft_consensus.cc:2272] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:08.224534 10435 ts_tablet_manager.cc:1916] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:08.224753 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:08.251058 9340 raft_consensus.cc:359] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.251813 9340 raft_consensus.cc:385] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:08.252076 9340 raft_consensus.cc:740] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:08.252693 9340 consensus_queue.cc:260] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.238173 9339 tablet_bootstrap.cc:654] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:08.328025 9006 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:56:08.329116 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:08.243423 10395 raft_consensus.cc:515] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.331280 9071 raft_consensus.cc:2393] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:08.331450 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:08.333420 9006 leader_election.cc:343] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:08.259353 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 215) took 9908 ms. Trace:
I20260926 01:56:08.334781 9711 rpcz_store.cc:276] 0926 01:55:58.351316 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:58.351456 (+ 140us) service_pool.cc:224] Handling call
0926 01:56:08.219438 (+9867982us) ts_tablet_manager.cc:1151] Deleting tablet b4c64a7619c4452f94b8d8b4f8bbbcf8
0926 01:56:08.258921 (+ 39483us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:08.259280 (+ 359us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9775451}
I20260926 01:56:08.337998 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 221) took 8938 ms. Trace:
I20260926 01:56:08.338343 9711 rpcz_store.cc:276] 0926 01:55:59.400082 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.400225 (+ 143us) service_pool.cc:224] Handling call
0926 01:56:08.337555 (+8937330us) ts_tablet_manager.cc:1151] Deleting tablet 030c41a9718d47ddaa33382b20a60dab
0926 01:56:08.337613 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:08.337936 (+ 323us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8935853}
I20260926 01:56:08.338816 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:08.340772 9006 leader_election.cc:343] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:08.345711 10435 ts_tablet_manager.cc:1929] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:08.369851 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:08.370739 9217 raft_consensus.cc:3060] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.380291 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:08.381995 10495 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:08.382536 10495 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:08.385777 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:08.386358 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 7.
I20260926 01:56:08.387771 9923 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters:
I20260926 01:56:08.391355 9711 tablet_replica.cc:333] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:08.393828 9711 raft_consensus.cc:2243] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:08.394381 9711 raft_consensus.cc:2272] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:08.419600 10487 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:08.420048 9711 ts_tablet_manager.cc:1916] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:08.420261 10487 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:08.422574 9341 raft_consensus.cc:359] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.423687 9341 raft_consensus.cc:385] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:08.424078 9341 raft_consensus.cc:740] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:08.358724 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:08.432202 8897 raft_consensus.cc:2468] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:08.363355 10475 tablet.cc:1751] T a09351c3db384b4383ca5ff0eaca982c P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260926 01:56:08.365016 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:08.433871 8905 raft_consensus.cc:3060] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.435001 8843 leader_election.cc:304] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:08.389916 9340 ts_tablet_manager.cc:1434] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.170s user 0.006s sys 0.001s
I20260926 01:56:08.437711 9340 tablet_bootstrap.cc:492] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:08.440006 10495 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:08.440605 10495 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260926 01:56:08.440927 10495 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:08.441267 10495 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 7 FOLLOWER]: Advancing to term 8
I20260926 01:56:08.417564 9339 tablet_bootstrap.cc:492] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:08.442904 9339 ts_tablet_manager.cc:1403] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.273s user 0.006s sys 0.007s
I20260926 01:56:08.399701 10395 leader_election.cc:290] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:08.445165 9923 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.189:34163, user_credentials={real_user=slave}} blocked reactor thread for 57435us
I20260926 01:56:08.446130 10340 raft_consensus.cc:2804] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:08.446710 10340 raft_consensus.cc:493] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:08.447041 10340 raft_consensus.cc:3060] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.448351 8843 leader_election.cc:304] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:08.452131 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:08.455773 10435 log.cc:1199] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:08.462568 10487 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:08.464218 10435 ts_tablet_manager.cc:1950] T 1743e863cdd84f3f8ca413fa81cd1914 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:08.462080 8905 raft_consensus.cc:2468] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:08.468022 10468 consensus_queue.cc:1048] T 0d068e93fd374a6091d7473db069a7fa P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:08.454756 10340 raft_consensus.cc:515] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.468852 10343 raft_consensus.cc:2749] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:08.470212 10340 leader_election.cc:290] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:08.476863 8843 leader_election.cc:304] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:08.513441 10495 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:08.524828 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:56:08.531177 10495 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:08.537263 10395 raft_consensus.cc:493] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:08.537837 10395 raft_consensus.cc:515] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.524073 9339 raft_consensus.cc:359] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.560073 9339 raft_consensus.cc:385] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:08.571659 9341 consensus_queue.cc:260] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.572260 9339 raft_consensus.cc:740] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:08.573799 9341 ts_tablet_manager.cc:1434] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.385s user 0.010s sys 0.017s
I20260926 01:56:08.575634 9341 tablet_bootstrap.cc:492] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:08.591627 10395 leader_election.cc:290] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:08.465919 9217 raft_consensus.cc:2468] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:08.609994 9341 tablet_bootstrap.cc:654] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:08.611490 10127 consensus_queue.cc:1048] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:08.628396 9003 leader_election.cc:343] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:56:08.637737 10478 consensus_queue.cc:1048] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:08.504004 9340 tablet_bootstrap.cc:654] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:08.512476 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e57300d2bcf4e0496ed13a4e7d967f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:08.628872 9003 leader_election.cc:304] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:08.650527 9339 consensus_queue.cc:260] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.653514 10355 raft_consensus.cc:2749] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:08.659479 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 182680us
I20260926 01:56:08.665021 9339 ts_tablet_manager.cc:1434] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.221s user 0.008s sys 0.001s
I20260926 01:56:08.665148 8905 raft_consensus.cc:2393] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:08.665650 10468 raft_consensus.cc:2749] T 7e57300d2bcf4e0496ed13a4e7d967f3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:56:08.665961 10435 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 285) took 1192 ms. Trace:
I20260926 01:56:08.666323 10435 rpcz_store.cc:276] 0926 01:56:07.473460 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.473586 (+ 126us) service_pool.cc:224] Handling call
0926 01:56:08.208643 (+735057us) ts_tablet_manager.cc:1151] Deleting tablet 1743e863cdd84f3f8ca413fa81cd1914
0926 01:56:08.208684 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:08.269030 (+ 60346us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.345556 (+ 76526us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.665891 (+320335us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":728661}
I20260926 01:56:08.667662 10435 tablet_replica.cc:333] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:08.668977 10435 raft_consensus.cc:2243] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:08.670018 10435 raft_consensus.cc:2272] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:08.672611 10435 ts_tablet_manager.cc:1916] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:08.682993 9003 leader_election.cc:304] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:08.687584 9006 leader_election.cc:304] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:08.692830 10355 raft_consensus.cc:2804] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:08.693279 10355 raft_consensus.cc:493] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:08.693612 10355 raft_consensus.cc:3060] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.699267 10502 raft_consensus.cc:2804] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:08.707033 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:08.714749 9003 leader_election.cc:304] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:08.717607 9339 tablet_bootstrap.cc:492] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:08.718730 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 57304.2us
I20260926 01:56:08.715157 10502 raft_consensus.cc:697] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:08.720834 10475 tablet.cc:1751] T d0bb5090e4874ae188b6d1ffe8649b4e P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:08.721786 10502 consensus_queue.cc:237] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.726483 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:08.706185 10355 raft_consensus.cc:515] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:08.739269 9006 leader_election.cc:343] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:08.734313 10355 leader_election.cc:290] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:08.756323 10503 raft_consensus.cc:2804] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:08.757246 10503 raft_consensus.cc:697] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:08.758250 10503 consensus_queue.cc:237] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.763682 9341 tablet_bootstrap.cc:492] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:08.767663 9341 ts_tablet_manager.cc:1403] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.192s user 0.020s sys 0.008s
I20260926 01:56:08.768174 10475 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:58226 (request call id 289) took 1121 ms. Trace:
I20260926 01:56:08.769305 9339 tablet_bootstrap.cc:654] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:08.768471 9340 tablet_bootstrap.cc:492] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:08.774880 9340 ts_tablet_manager.cc:1403] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.338s user 0.011s sys 0.012s
I20260926 01:56:08.777710 9340 raft_consensus.cc:359] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.770491 10475 rpcz_store.cc:276] 0926 01:56:07.646953 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.649570 (+ 2617us) service_pool.cc:224] Handling call
0926 01:56:08.768058 (+1118488us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:07.703143 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0926 01:56:07.703553 (+ 410us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0926 01:56:07.703584 (+ 31us) alter_schema_op.cc:66] Acquired schema lock
0926 01:56:07.703659 (+ 75us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0926 01:56:07.703824 (+ 165us) alter_schema_op.cc:128] START. Timestamp: 1790387767703706
0926 01:56:07.703841 (+ 17us) op_driver.cc:348] REPLICATION: starting
0926 01:56:07.704494 (+ 653us) log.cc:844] Serialized 173 byte log entry
0926 01:56:08.720429 (+1015935us) op_driver.cc:464] REPLICATION: finished
0926 01:56:08.720739 (+ 310us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0926 01:56:08.722172 (+ 1433us) log.cc:844] Serialized 5 byte log entry
0926 01:56:08.731101 (+ 8929us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.748355 (+ 17254us) log.cc:844] Serialized 14 byte log entry
0926 01:56:08.766357 (+ 18002us) alter_schema_op.cc:73] Released schema lock
0926 01:56:08.766397 (+ 40us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":125,"prepare.queue_time_us":211,"prepare.run_cpu_time_us":9449,"prepare.run_wall_time_us":67770,"replication_time_us":1016433,"thread_start_us":65867,"threads_started":1,"wal-append.queue_time_us":59818}]]}
I20260926 01:56:08.782083 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:08.780169 9341 raft_consensus.cc:359] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.778630 9340 raft_consensus.cc:385] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:08.794706 9340 raft_consensus.cc:740] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:08.795789 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:08.801967 8905 raft_consensus.cc:2468] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:08.806589 9341 raft_consensus.cc:385] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:08.807165 9341 raft_consensus.cc:740] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:08.808133 9341 consensus_queue.cc:260] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:08.810411 9217 raft_consensus.cc:3060] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:08.814026 9985 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 8 FOLLOWER]: Rejecting Update request from peer ca25983598bc4c0688340d8e7bc0f6ba for earlier term 7. Current term is 8. Ops: []
I20260926 01:56:08.816138 10246 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 }, Status: INVALID_TERM, Last received: 7.357, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260926 01:56:08.823488 9340 consensus_queue.cc:260] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.828752 9217 raft_consensus.cc:2468] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:08.831105 9711 ts_tablet_manager.cc:1929] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:08.831493 9711 log.cc:1199] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/ff004cf835c34095a836ad683cec9256
I20260926 01:56:08.832136 9711 ts_tablet_manager.cc:1950] T ff004cf835c34095a836ad683cec9256 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:08.833626 9340 ts_tablet_manager.cc:1434] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.058s user 0.003s sys 0.002s
I20260926 01:56:08.834218 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 223) took 9056 ms. Trace:
I20260926 01:56:08.845094 9340 tablet_bootstrap.cc:492] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:08.850839 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:08.857440 8906 raft_consensus.cc:2393] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
W20260926 01:56:08.859447 8843 leader_election.cc:343] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:08.868427 9341 ts_tablet_manager.cc:1434] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.097s user 0.008s sys 0.002s
I20260926 01:56:08.873593 10511 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 7 LEADER]: Stepping down as leader of term 7
I20260926 01:56:08.842547 9006 leader_election.cc:304] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:08.862926 10508 tablet.cc:1751] T d0bb5090e4874ae188b6d1ffe8649b4e P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:08.865092 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05042bcc0a6e426d9e3e8b25d8d55540" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:08.884167 10511 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:56:08.893153 10511 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 7.357, Last appended by leader: 357, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:08.894767 9341 tablet_bootstrap.cc:492] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:08.903111 10511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 7 FOLLOWER]: Advancing to term 8
I20260926 01:56:08.879129 9339 tablet_bootstrap.cc:492] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:08.932698 9341 tablet_bootstrap.cc:654] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:08.919937 9339 ts_tablet_manager.cc:1403] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.250s user 0.009s sys 0.006s
I20260926 01:56:08.946192 10340 raft_consensus.cc:493] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:08.946851 10340 raft_consensus.cc:515] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:08.917599 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 75156.8us
I20260926 01:56:08.949057 10340 leader_election.cc:290] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:08.917040 9066 raft_consensus.cc:2393] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:08.950583 10503 raft_consensus.cc:2804] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:08.951368 10503 raft_consensus.cc:697] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:08.951289 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:08.952268 10503 consensus_queue.cc:237] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:08.953087 8843 leader_election.cc:343] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:08.918879 9711 rpcz_store.cc:276] 0926 01:55:59.778194 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:55:59.778315 (+ 121us) service_pool.cc:224] Handling call
0926 01:56:08.391149 (+8612834us) ts_tablet_manager.cc:1151] Deleting tablet ff004cf835c34095a836ad683cec9256
0926 01:56:08.391211 (+ 62us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:08.674698 (+283487us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.830967 (+156269us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:08.834157 (+ 3190us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8612180}
I20260926 01:56:08.955163 10516 tablet.cc:1751] T d0bb5090e4874ae188b6d1ffe8649b4e P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:08.902398 9340 tablet_bootstrap.cc:654] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:08.979750 9339 raft_consensus.cc:359] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:08.980497 9339 raft_consensus.cc:385] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:08.980768 9339 raft_consensus.cc:740] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:08.987972 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 224) took 8723 ms. Trace:
I20260926 01:56:08.988353 9711 rpcz_store.cc:276] 0926 01:56:00.264266 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.264397 (+ 131us) service_pool.cc:224] Handling call
0926 01:56:08.954318 (+8689921us) ts_tablet_manager.cc:1151] Deleting tablet 2156ef6b24c04ea38181832e7f18ff49
0926 01:56:08.987526 (+ 33208us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:08.987897 (+ 371us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8689304}
I20260926 01:56:08.989742 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 225) took 8668 ms. Trace:
I20260926 01:56:08.989451 9341 tablet_bootstrap.cc:492] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:08.990036 9711 rpcz_store.cc:276] 0926 01:56:00.321537 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.322508 (+ 971us) service_pool.cc:224] Handling call
0926 01:56:08.989352 (+8666844us) ts_tablet_manager.cc:1151] Deleting tablet 2156ef6b24c04ea38181832e7f18ff49
0926 01:56:08.989408 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:08.989685 (+ 277us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8666207}
I20260926 01:56:08.991052 9711 tablet_replica.cc:333] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:08.991873 9711 raft_consensus.cc:2243] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:08.996470 9341 ts_tablet_manager.cc:1403] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.100s user 0.018s sys 0.000s
I20260926 01:56:09.009444 9339 consensus_queue.cc:260] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.048281 9341 raft_consensus.cc:359] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:09.038552 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 53898.9us
W20260926 01:56:09.036418 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 62596.6us
I20260926 01:56:09.052464 9003 leader_election.cc:304] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:09.053246 10502 raft_consensus.cc:2804] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:09.053575 10502 raft_consensus.cc:493] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:09.053900 10502 raft_consensus.cc:3060] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.062839 10502 raft_consensus.cc:515] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.065129 10502 leader_election.cc:290] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.037065 9711 raft_consensus.cc:2272] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:09.068559 9339 ts_tablet_manager.cc:1434] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.123s user 0.004s sys 0.004s
I20260926 01:56:09.069401 10435 ts_tablet_manager.cc:1929] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260926 01:56:09.075291 9003 consensus_peers.cc:597] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:09.066690 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:09.081914 9006 leader_election.cc:343] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.083025 9341 raft_consensus.cc:385] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.083395 9341 raft_consensus.cc:740] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.084187 9341 consensus_queue.cc:260] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.086886 9711 ts_tablet_manager.cc:1916] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:09.101290 9341 ts_tablet_manager.cc:1434] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.088s user 0.008s sys 0.000s
I20260926 01:56:09.110690 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:09.125691 10435 log.cc:1199] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:09.126802 10435 ts_tablet_manager.cc:1950] T 17b8b0d54f5645e5963f1de217f94c84 P 49bd320bf73f4252804ec867ec077510: Deleting consensus metadata
I20260926 01:56:09.085541 9340 tablet_bootstrap.cc:492] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:09.098012 8843 leader_election.cc:304] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:09.105278 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:09.127920 9339 tablet_bootstrap.cc:492] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:09.127907 8897 raft_consensus.cc:3060] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.129479 10464 raft_consensus.cc:2749] T 05042bcc0a6e426d9e3e8b25d8d55540 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:56:09.133630 9340 ts_tablet_manager.cc:1403] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.289s user 0.012s sys 0.001s
I20260926 01:56:09.136332 9340 raft_consensus.cc:359] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.137120 9340 raft_consensus.cc:385] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.137437 9340 raft_consensus.cc:740] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.138833 9339 tablet_bootstrap.cc:654] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:09.138908 8599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.704s (SENT)
W20260926 01:56:09.142504 8843 leader_election.cc:343] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.142988 8843 leader_election.cc:304] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:09.143707 10340 raft_consensus.cc:2749] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:09.149596 9341 tablet_bootstrap.cc:492] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:09.165791 9341 tablet_bootstrap.cc:654] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.190059 10502 raft_consensus.cc:493] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:09.190618 10502 raft_consensus.cc:515] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.192762 10502 leader_election.cc:290] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.138207 9340 consensus_queue.cc:260] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.196796 9340 ts_tablet_manager.cc:1434] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20260926 01:56:09.197872 9340 tablet_bootstrap.cc:492] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:09.198976 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b778c95327fc477ca210cb3c983665c1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:09.199616 9218 raft_consensus.cc:2468] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:09.200925 9006 leader_election.cc:304] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:09.201408 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b778c95327fc477ca210cb3c983665c1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:09.201838 10478 raft_consensus.cc:2804] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:09.202113 8905 raft_consensus.cc:2468] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:09.202200 10478 raft_consensus.cc:493] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:09.202524 10478 raft_consensus.cc:3060] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.222522 10340 raft_consensus.cc:493] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:09.223342 10340 raft_consensus.cc:515] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.226518 10502 raft_consensus.cc:493] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc00feec3f564608851f0112e97ab532)
I20260926 01:56:09.229724 10502 raft_consensus.cc:515] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:09.231096 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 92311.8us
I20260926 01:56:09.231912 10502 leader_election.cc:290] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.214291 10478 raft_consensus.cc:515] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:09.237811 9006 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 3571967749844fd7859cf179a93cf403
I20260926 01:56:09.243832 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b778c95327fc477ca210cb3c983665c1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:09.244510 8899 raft_consensus.cc:3060] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.246747 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b778c95327fc477ca210cb3c983665c1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:09.247421 9218 raft_consensus.cc:3060] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.249534 10478 leader_election.cc:290] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:09.269030 9003 leader_election.cc:343] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 3571967749844fd7859cf179a93cf403
I20260926 01:56:09.269219 8897 raft_consensus.cc:2468] T 0c6cccff3de64f3384ce58bb84c43487 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:09.269712 9003 leader_election.cc:304] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:09.271294 9003 leader_election.cc:304] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:09.272089 10502 raft_consensus.cc:2749] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:09.272711 10502 raft_consensus.cc:2804] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:09.273114 10502 raft_consensus.cc:697] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:09.273923 10502 consensus_queue.cc:237] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.275068 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:09.276795 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:09.277400 9066 raft_consensus.cc:2468] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
W20260926 01:56:09.278040 8843 leader_election.cc:343] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.280087 8843 leader_election.cc:304] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:09.281212 10343 raft_consensus.cc:2804] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:09.281541 10343 raft_consensus.cc:493] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:09.281857 10343 raft_consensus.cc:3060] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.289372 9339 tablet_bootstrap.cc:492] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:09.289860 9339 ts_tablet_manager.cc:1403] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.006s
I20260926 01:56:09.292815 9339 raft_consensus.cc:359] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.293500 9339 raft_consensus.cc:385] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.293473 10435 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 286) took 1818 ms. Trace:
I20260926 01:56:09.293762 9339 raft_consensus.cc:740] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.294484 9339 consensus_queue.cc:260] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.296567 9339 ts_tablet_manager.cc:1434] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 01:56:09.297561 9339 tablet_bootstrap.cc:492] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:09.302021 10511 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:09.302546 10511 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:09.304701 10511 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:09.305881 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 9 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:09.305946 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 9 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:09.306509 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 8.
I20260926 01:56:09.306576 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 7.
I20260926 01:56:09.307930 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:56:09.309302 10511 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260926 01:56:09.309710 10511 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:09.310107 10511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 8 FOLLOWER]: Advancing to term 9
I20260926 01:56:09.313112 10435 rpcz_store.cc:276] 0926 01:56:07.474879 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.485303 (+ 10424us) service_pool.cc:224] Handling call
0926 01:56:08.667472 (+1182169us) ts_tablet_manager.cc:1151] Deleting tablet 17b8b0d54f5645e5963f1de217f94c84
0926 01:56:08.667519 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:08.871083 (+203564us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:09.069247 (+198164us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:09.293401 (+224154us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171254}
I20260926 01:56:09.314882 10435 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58226 (request call id 284) took 1853 ms. Trace:
I20260926 01:56:09.315335 10435 rpcz_store.cc:276] 0926 01:56:07.461126 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.465681 (+ 4555us) service_pool.cc:224] Handling call
0926 01:56:09.314556 (+1848875us) ts_tablet_manager.cc:1151] Deleting tablet 72f666b1c069435b8ff0a3fae9f1e873
0926 01:56:09.314609 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:09.314816 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85,"tablet-delete.queue_time_us":1818317}
I20260926 01:56:09.318243 9341 tablet_bootstrap.cc:492] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:09.318742 9341 ts_tablet_manager.cc:1403] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.169s user 0.006s sys 0.012s
I20260926 01:56:09.321460 9341 raft_consensus.cc:359] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.322146 9341 raft_consensus.cc:385] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.330629 9339 tablet_bootstrap.cc:654] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.322434 9341 raft_consensus.cc:740] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.333549 9341 consensus_queue.cc:260] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.341887 10496 raft_consensus.cc:493] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:09.345131 10496 raft_consensus.cc:515] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.348183 9340 tablet_bootstrap.cc:654] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.350086 10496 leader_election.cc:290] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:09.350651 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:09.353950 10511 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:09.359298 9341 ts_tablet_manager.cc:1434] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.040s user 0.001s sys 0.006s
I20260926 01:56:09.368739 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 9 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:56:09.369334 8624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 7 FOLLOWER]: Advancing to term 9
I20260926 01:56:09.378293 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 9.
I20260926 01:56:09.363996 9341 tablet_bootstrap.cc:492] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:09.380229 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:56:09.389653 9339 tablet_bootstrap.cc:492] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:09.351462 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accc62f46ce24a4d97743dca05e65f51" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:09.374795 9218 raft_consensus.cc:2468] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:09.393283 8897 raft_consensus.cc:2393] T accc62f46ce24a4d97743dca05e65f51 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:09.372804 10340 leader_election.cc:290] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.292970 10343 raft_consensus.cc:515] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.394604 9152 leader_election.cc:304] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:09.396278 10343 leader_election.cc:290] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.396827 10496 raft_consensus.cc:3060] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.398797 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 9 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849"
I20260926 01:56:09.399423 9985 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 8 FOLLOWER]: Advancing to term 9
I20260926 01:56:09.406718 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:09.411091 8843 leader_election.cc:343] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.413316 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 9.
I20260926 01:56:09.417515 10511 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 9 election: Requested vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:09.406718 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:09.426779 9066 raft_consensus.cc:3060] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.416918 9006 leader_election.cc:304] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:09.431612 10502 raft_consensus.cc:2804] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:09.432113 10502 raft_consensus.cc:697] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:09.432981 10502 consensus_queue.cc:237] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.434935 10496 raft_consensus.cc:2749] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:09.439591 9340 tablet_bootstrap.cc:492] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:09.389330 8899 raft_consensus.cc:2468] T b778c95327fc477ca210cb3c983665c1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:09.440188 9340 ts_tablet_manager.cc:1403] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.243s user 0.009s sys 0.003s
I20260926 01:56:09.449090 10478 raft_consensus.cc:493] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:09.449074 9340 raft_consensus.cc:359] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.449764 9340 raft_consensus.cc:385] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.449575 10478 raft_consensus.cc:515] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.450006 9340 raft_consensus.cc:740] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.451781 10478 leader_election.cc:290] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.444850 10526 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 9 FOLLOWER]: Leader election won for term 9
I20260926 01:56:09.455579 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:09.456487 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:09.457685 8899 raft_consensus.cc:2468] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:09.458822 9003 leader_election.cc:304] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:56:09.459744 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8e0e6ede1ce4487b08bfd5743e3656a consensus state has no leader
I20260926 01:56:09.460687 10478 raft_consensus.cc:2804] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:09.461149 10478 raft_consensus.cc:493] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:09.461556 10478 raft_consensus.cc:3060] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:09.463109 9006 leader_election.cc:343] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:09.466638 8526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.189:34163, user_credentials={real_user=slave}} blocked reactor thread for 86509.3us
I20260926 01:56:09.469249 9066 raft_consensus.cc:2468] T e76071c66b1d4635ac1a28110eb684fa P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:09.474402 9340 consensus_queue.cc:260] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:09.475271 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.516s (SENT)
I20260926 01:56:09.478183 9340 ts_tablet_manager.cc:1434] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.037s user 0.000s sys 0.007s
I20260926 01:56:09.479245 9340 tablet_bootstrap.cc:492] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:09.476742 9339 ts_tablet_manager.cc:1403] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.179s user 0.008s sys 0.004s
I20260926 01:56:09.492360 8843 leader_election.cc:304] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:09.493805 10343 raft_consensus.cc:2804] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:09.494443 10343 raft_consensus.cc:697] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:09.495330 10343 consensus_queue.cc:237] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.495694 9339 raft_consensus.cc:359] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.496397 9339 raft_consensus.cc:385] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.496655 9339 raft_consensus.cc:740] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.511057 10526 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 9 LEADER]: Becoming Leader. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:56:09.525671 9340 tablet_bootstrap.cc:654] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.534399 10502 consensus_queue.cc:1048] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:09.560535 9340 tablet_bootstrap.cc:492] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:09.564044 10478 raft_consensus.cc:515] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.564725 9340 ts_tablet_manager.cc:1403] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.008s
I20260926 01:56:09.570700 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:09.572933 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:09.573577 8899 raft_consensus.cc:3060] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:09.574746 10530 log.cc:927] Time spent T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
W20260926 01:56:09.575591 9006 leader_election.cc:343] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.595120 10355 raft_consensus.cc:493] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:09.595643 10355 raft_consensus.cc:515] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.597882 10355 leader_election.cc:290] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.601118 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:09.601766 8897 raft_consensus.cc:2468] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:09.601792 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:09.603066 9003 leader_election.cc:304] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:09.604029 10355 raft_consensus.cc:2804] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:09.604478 10355 raft_consensus.cc:493] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:09.604812 10355 raft_consensus.cc:3060] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.611289 9340 raft_consensus.cc:359] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:09.619606 9006 leader_election.cc:343] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.621956 9341 tablet_bootstrap.cc:700] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532: Time spent opening tablet: real 0.112s user 0.003s sys 0.000s
I20260926 01:56:09.622383 9340 raft_consensus.cc:385] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.622851 9341 tablet_bootstrap.cc:654] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.623179 9340 raft_consensus.cc:740] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.613032 8899 raft_consensus.cc:2468] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:09.626652 9340 consensus_queue.cc:260] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.627635 9003 leader_election.cc:304] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:09.629232 10503 raft_consensus.cc:2804] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:09.629964 10503 raft_consensus.cc:697] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:09.631057 9340 ts_tablet_manager.cc:1434] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20260926 01:56:09.631417 10503 consensus_queue.cc:237] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.636380 10127 consensus_queue.cc:1048] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:09.637756 10127 consensus_queue.cc:1048] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:09.497365 9339 consensus_queue.cc:260] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:09.662961 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.568s (SENT)
I20260926 01:56:09.663486 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:09.564098 10526 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 7.357, Last appended by leader: 357, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:09.674353 10478 leader_election.cc:290] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.648469 9339 ts_tablet_manager.cc:1434] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.158s user 0.005s sys 0.001s
I20260926 01:56:09.677798 9711 ts_tablet_manager.cc:1929] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:56:09.678228 9711 log.cc:1199] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/f100eb5dc9a8452e943d4cce5284bc8a
I20260926 01:56:09.679786 9339 tablet_bootstrap.cc:492] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:09.705008 10538 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260926 01:56:09.707535 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 44631.2us
I20260926 01:56:09.710211 9341 tablet_bootstrap.cc:492] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:09.718160 9340 tablet_bootstrap.cc:492] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:09.725262 9340 tablet_bootstrap.cc:654] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.727768 10355 raft_consensus.cc:515] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.730705 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:09.731330 8897 raft_consensus.cc:3060] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:09.740372 8897 raft_consensus.cc:2468] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:09.743178 9003 leader_election.cc:304] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:09.744030 10502 raft_consensus.cc:2804] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:09.744483 10502 raft_consensus.cc:697] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:09.745319 10502 consensus_queue.cc:237] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.746570 9340 tablet_bootstrap.cc:492] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:09.747107 9340 ts_tablet_manager.cc:1403] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.004s
I20260926 01:56:09.754411 9341 ts_tablet_manager.cc:1403] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.391s user 0.008s sys 0.004s
I20260926 01:56:09.743153 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:09.765933 9340 raft_consensus.cc:359] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.767228 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 9 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:09.767987 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:09.746089 9711 ts_tablet_manager.cc:1950] T f100eb5dc9a8452e943d4cce5284bc8a P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:09.771441 9340 raft_consensus.cc:385] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.772123 9340 raft_consensus.cc:740] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.773231 9340 consensus_queue.cc:260] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.778344 10503 consensus_queue.cc:1048] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:09.782312 9340 ts_tablet_manager.cc:1434] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
W20260926 01:56:09.785365 9006 leader_election.cc:343] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.787209 9340 tablet_bootstrap.cc:492] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:09.799124 10355 leader_election.cc:290] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:09.757231 9341 raft_consensus.cc:359] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.803013 9341 raft_consensus.cc:385] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.804137 9341 raft_consensus.cc:740] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.804903 9341 consensus_queue.cc:260] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:09.808413 9339 tablet_bootstrap.cc:654] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.825418 9340 tablet_bootstrap.cc:654] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.867064 9340 tablet_bootstrap.cc:492] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:09.867517 9340 ts_tablet_manager.cc:1403] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.002s
I20260926 01:56:09.818274 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 233) took 8957 ms. Trace:
I20260926 01:56:09.878482 9339 tablet_bootstrap.cc:492] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:09.883348 9340 raft_consensus.cc:359] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.884434 9339 ts_tablet_manager.cc:1403] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.199s user 0.009s sys 0.004s
I20260926 01:56:09.890614 9340 raft_consensus.cc:385] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.891033 9340 raft_consensus.cc:740] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.894214 9340 consensus_queue.cc:260] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.902005 9340 ts_tablet_manager.cc:1434] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260926 01:56:09.902285 9339 raft_consensus.cc:359] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.913475 9340 tablet_bootstrap.cc:492] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:09.909397 9339 raft_consensus.cc:385] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:09.925375 9341 ts_tablet_manager.cc:1434] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.171s user 0.003s sys 0.007s
I20260926 01:56:09.922627 9339 raft_consensus.cc:740] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:09.927271 9339 consensus_queue.cc:260] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.933506 9339 ts_tablet_manager.cc:1434] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.049s user 0.004s sys 0.003s
I20260926 01:56:09.949137 9340 tablet_bootstrap.cc:654] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:09.960033 10355 consensus_queue.cc:1048] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:56:09.980278 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.704s (SENT)
I20260926 01:56:09.980719 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:09.983129 10340 raft_consensus.cc:493] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:09.983650 10340 raft_consensus.cc:515] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:09.985659 10340 leader_election.cc:290] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:09.988706 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:09.989315 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:09.990217 8843 leader_election.cc:343] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:09.991448 8843 leader_election.cc:343] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:09.992785 8843 leader_election.cc:304] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:09.994818 10340 raft_consensus.cc:2749] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.008833 9340 tablet_bootstrap.cc:492] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:10.009423 9340 ts_tablet_manager.cc:1403] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.002s
I20260926 01:56:10.005597 10554 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260926 01:56:09.875736 9711 rpcz_store.cc:276] 0926 01:56:00.859549 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.859686 (+ 137us) service_pool.cc:224] Handling call
0926 01:56:08.990896 (+8131210us) ts_tablet_manager.cc:1151] Deleting tablet f100eb5dc9a8452e943d4cce5284bc8a
0926 01:56:08.990945 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:09.472143 (+481198us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:09.677649 (+205506us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:09.816528 (+138879us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44262,"tablet-delete.queue_time_us":8130613}
I20260926 01:56:10.012987 9340 raft_consensus.cc:359] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.019798 9340 raft_consensus.cc:385] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.023180 9340 raft_consensus.cc:740] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.023908 9340 consensus_queue.cc:260] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.031782 10502 consensus_queue.cc:1048] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:10.041431 9340 ts_tablet_manager.cc:1434] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260926 01:56:10.056773 9340 tablet_bootstrap.cc:492] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:10.067927 9340 tablet_bootstrap.cc:654] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.091367 9339 tablet_bootstrap.cc:492] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:10.094055 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 232) took 9234 ms. Trace:
I20260926 01:56:10.106968 9339 tablet_bootstrap.cc:654] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.115846 10468 raft_consensus.cc:493] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.103502 9711 rpcz_store.cc:276] 0926 01:56:00.859255 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:00.862562 (+ 3307us) service_pool.cc:224] Handling call
0926 01:56:10.089834 (+9227272us) ts_tablet_manager.cc:1151] Deleting tablet 75d4598596ff49378fc3536fff0ac1b1
0926 01:56:10.093701 (+ 3867us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:10.093977 (+ 276us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9226650}
I20260926 01:56:10.116954 9711 tablet_replica.cc:333] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:10.116580 10468 raft_consensus.cc:515] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.117764 9711 raft_consensus.cc:2243] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:10.118933 10468 leader_election.cc:290] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.125349 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cbf646569e6497dac90e308f0681372" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.126044 9067 raft_consensus.cc:2468] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:10.137838 8843 leader_election.cc:304] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:10.138669 10551 raft_consensus.cc:493] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.138983 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cbf646569e6497dac90e308f0681372" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:10.141414 9341 tablet_bootstrap.cc:492] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:10.142599 8843 leader_election.cc:343] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.143613 10468 raft_consensus.cc:2804] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:10.144096 10468 raft_consensus.cc:493] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:10.144508 10468 raft_consensus.cc:3060] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.144743 9339 tablet_bootstrap.cc:492] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:10.145308 9339 ts_tablet_manager.cc:1403] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.000s
I20260926 01:56:10.146744 10496 raft_consensus.cc:493] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.148027 10496 raft_consensus.cc:515] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.139151 10551 raft_consensus.cc:515] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.159454 10503 consensus_queue.cc:1048] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:10.164119 9985 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 7 index: 357. Preceding OpId from leader: term: 9 index: 358. (index mismatch)
I20260926 01:56:10.167400 10511 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260926 01:56:10.171141 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:10.171933 8899 raft_consensus.cc:2393] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:10.180387 9339 raft_consensus.cc:359] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.181061 9339 raft_consensus.cc:385] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.181344 9339 raft_consensus.cc:740] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.118299 9711 raft_consensus.cc:2272] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:10.187409 9711 ts_tablet_manager.cc:1916] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:10.187768 10468 raft_consensus.cc:515] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.191888 10551 leader_election.cc:290] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.193275 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4c64a7619c4452f94b8d8b4f8bbbcf8" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.196461 9341 tablet_bootstrap.cc:654] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.197055 10538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 9 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:10.197767 10538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:10.211172 10538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:10.211930 10538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:10.219666 9155 leader_election.cc:304] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:10.222020 10546 raft_consensus.cc:3060] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.229537 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:10.233103 10496 leader_election.cc:290] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.240448 9340 tablet_bootstrap.cc:492] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:10.240885 9340 ts_tablet_manager.cc:1403] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.184s user 0.006s sys 0.003s
I20260926 01:56:10.243741 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6cccff3de64f3384ce58bb84c43487" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.245330 8624 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 7 index: 357. Preceding OpId from leader: term: 9 index: 358. (index mismatch)
I20260926 01:56:10.245775 9155 leader_election.cc:304] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:10.246876 10551 raft_consensus.cc:2749] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.248651 9340 raft_consensus.cc:359] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.249405 9340 raft_consensus.cc:385] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.249233 10511 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260926 01:56:10.249706 9340 raft_consensus.cc:740] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.250674 9340 consensus_queue.cc:260] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.255798 10546 raft_consensus.cc:2749] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.258190 9340 ts_tablet_manager.cc:1434] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260926 01:56:10.259303 9340 tablet_bootstrap.cc:492] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:10.277182 10355 consensus_queue.cc:1048] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:10.181937 9339 consensus_queue.cc:260] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.295383 9218 raft_consensus.cc:3060] T 0c6cccff3de64f3384ce58bb84c43487 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.303467 9339 ts_tablet_manager.cc:1434] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.158s user 0.009s sys 0.000s
I20260926 01:56:10.304764 10478 consensus_queue.cc:1048] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:10.305733 9339 tablet_bootstrap.cc:492] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:10.312217 10468 leader_election.cc:290] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.318079 9341 tablet_bootstrap.cc:492] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:10.327869 9341 ts_tablet_manager.cc:1403] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.187s user 0.011s sys 0.010s
I20260926 01:56:10.340229 10355 consensus_queue.cc:1048] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:56:10.345355 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 46722.5us
I20260926 01:56:10.350148 9340 tablet_bootstrap.cc:654] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.352833 9341 raft_consensus.cc:359] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.353545 9341 raft_consensus.cc:385] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.353806 9341 raft_consensus.cc:740] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.357803 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cbf646569e6497dac90e308f0681372" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:10.358390 9067 raft_consensus.cc:3060] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.360337 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cbf646569e6497dac90e308f0681372" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:10.363293 9341 consensus_queue.cc:260] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.366349 10340 raft_consensus.cc:493] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.366884 10340 raft_consensus.cc:515] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.367061 10526 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:10.368942 10340 leader_election.cc:290] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.367772 10526 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:10.372164 9067 raft_consensus.cc:2468] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:10.376281 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.377681 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:10.382930 10355 raft_consensus.cc:493] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:10.383496 10355 raft_consensus.cc:515] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.387493 10355 leader_election.cc:290] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:10.390489 9003 leader_election.cc:343] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 92896db04df346bf91f732b1ce89cd23
I20260926 01:56:10.413694 10554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 9 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:10.414417 10554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:56:10.419531 8843 leader_election.cc:343] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:10.420217 8843 leader_election.cc:343] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.422750 8843 leader_election.cc:304] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:10.423461 10340 raft_consensus.cc:2804] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:10.423903 10340 raft_consensus.cc:697] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:10.424669 10340 consensus_queue.cc:237] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:10.435344 8843 leader_election.cc:343] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.436079 8843 leader_election.cc:304] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:10.447198 9339 tablet_bootstrap.cc:700] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2: Time spent opening tablet: real 0.105s user 0.003s sys 0.000s
I20260926 01:56:10.447957 9339 tablet_bootstrap.cc:654] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.452585 10570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:10.453552 10570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:10.454005 10340 raft_consensus.cc:2749] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.455964 9341 ts_tablet_manager.cc:1434] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.128s user 0.006s sys 0.001s
I20260926 01:56:10.460584 9340 tablet_bootstrap.cc:492] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:10.461074 9340 ts_tablet_manager.cc:1403] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.202s user 0.012s sys 0.003s
I20260926 01:56:10.460760 9711 ts_tablet_manager.cc:1929] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:10.465483 9711 log.cc:1199] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/3571967749844fd7859cf179a93cf403
I20260926 01:56:10.466717 9711 ts_tablet_manager.cc:1950] T 3571967749844fd7859cf179a93cf403 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:10.470618 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 237) took 9413 ms. Trace:
I20260926 01:56:10.471665 9340 raft_consensus.cc:359] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.472324 9340 raft_consensus.cc:385] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.472584 9340 raft_consensus.cc:740] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.473197 9340 consensus_queue.cc:260] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.475165 9340 ts_tablet_manager.cc:1434] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260926 01:56:10.472930 9711 rpcz_store.cc:276] 0926 01:56:01.057140 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.057245 (+ 105us) service_pool.cc:224] Handling call
0926 01:56:10.116751 (+9059506us) ts_tablet_manager.cc:1151] Deleting tablet 3571967749844fd7859cf179a93cf403
0926 01:56:10.116830 (+ 79us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:10.315880 (+199050us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:10.460588 (+144708us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:10.470553 (+ 9965us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9058909}
I20260926 01:56:10.489718 9341 tablet_bootstrap.cc:492] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:10.495921 9341 tablet_bootstrap.cc:654] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.493952 9339 tablet_bootstrap.cc:492] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:10.501639 9339 ts_tablet_manager.cc:1403] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.196s user 0.008s sys 0.006s
W20260926 01:56:10.502593 9006 leader_election.cc:343] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 92896db04df346bf91f732b1ce89cd23
I20260926 01:56:10.503046 9006 leader_election.cc:304] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:10.504040 10355 raft_consensus.cc:2749] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:10.504215 9339 raft_consensus.cc:359] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.504873 9339 raft_consensus.cc:385] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.505113 9339 raft_consensus.cc:740] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.505844 9339 consensus_queue.cc:260] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.515722 9339 ts_tablet_manager.cc:1434] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
W20260926 01:56:10.525250 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 82992us
I20260926 01:56:10.546674 9711 tablet_replica.cc:333] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:10.548827 9711 raft_consensus.cc:2243] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:10.550035 9711 raft_consensus.cc:2272] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:10.559285 9339 tablet_bootstrap.cc:492] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:10.560004 10503 raft_consensus.cc:493] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.560703 9711 ts_tablet_manager.cc:1916] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:10.560969 10503 raft_consensus.cc:515] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.563019 10503 leader_election.cc:290] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:10.564785 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:10.565323 9340 tablet_bootstrap.cc:492] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:10.570791 9003 leader_election.cc:343] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.572773 9341 tablet_bootstrap.cc:492] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:10.573221 9341 ts_tablet_manager.cc:1403] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.004s
I20260926 01:56:10.575904 9340 tablet_bootstrap.cc:654] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.577128 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:10.575902 9341 raft_consensus.cc:359] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.579959 9341 raft_consensus.cc:385] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.580300 9341 raft_consensus.cc:740] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.582415 9339 tablet_bootstrap.cc:654] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.587569 9341 consensus_queue.cc:260] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.592443 10503 raft_consensus.cc:493] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.592947 10503 raft_consensus.cc:515] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.595482 10503 leader_election.cc:290] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:10.597015 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:10.597695 8897 raft_consensus.cc:2468] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:56:10.598053 9006 leader_election.cc:343] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.598498 9006 leader_election.cc:304] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:10.599618 9003 leader_election.cc:304] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:10.600309 10502 raft_consensus.cc:2749] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.600893 10502 raft_consensus.cc:2804] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:10.601173 10502 raft_consensus.cc:493] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:10.601466 10502 raft_consensus.cc:3060] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.607211 10355 raft_consensus.cc:493] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:10.607693 10355 raft_consensus.cc:515] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:10.609540 9341 ts_tablet_manager.cc:1434] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.036s user 0.007s sys 0.001s
I20260926 01:56:10.612566 9341 tablet_bootstrap.cc:492] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:10.618697 10355 leader_election.cc:290] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:10.621529 9003 leader_election.cc:343] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:10.623157 10502 raft_consensus.cc:515] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.625504 10503 raft_consensus.cc:493] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.625706 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:10.626303 8897 raft_consensus.cc:3060] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.625938 10503 raft_consensus.cc:515] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.629364 10503 leader_election.cc:290] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:10.630867 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:10.631516 8899 raft_consensus.cc:2468] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:10.633525 9003 leader_election.cc:304] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:10.634321 10355 raft_consensus.cc:2804] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:10.634670 10355 raft_consensus.cc:493] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:10.634974 10355 raft_consensus.cc:3060] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:10.641901 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4623456312284a698ae69d4054909647 (table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4623456312284a698ae69d4054909647 consensus state has no leader
W20260926 01:56:10.597337 8843 consensus_peers.cc:597] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:10.648373 10562 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:10.649147 10562 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:10.647847 8897 raft_consensus.cc:2468] T 5eef6a48e87e4c61a23058942a82bccc P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
W20260926 01:56:10.649039 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 51818us
I20260926 01:56:10.665776 9003 leader_election.cc:304] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:10.666648 10478 raft_consensus.cc:2804] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:10.672194 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:10.672080 10566 consensus_queue.cc:1048] T e76071c66b1d4635ac1a28110eb684fa P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:10.673925 10340 raft_consensus.cc:493] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.674365 10340 raft_consensus.cc:515] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.676297 10340 leader_election.cc:290] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.683401 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:10.685709 10502 leader_election.cc:290] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:10.687803 8843 leader_election.cc:343] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.689625 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.693156 9340 tablet_bootstrap.cc:492] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:10.693660 9340 ts_tablet_manager.cc:1403] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.001s
I20260926 01:56:10.696350 9340 raft_consensus.cc:359] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.697029 9340 raft_consensus.cc:385] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.698518 10551 raft_consensus.cc:493] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.699496 9711 ts_tablet_manager.cc:1929] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:10.699990 9711 log.cc:1199] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/943a990fa10948e8961c02e9999569f5
I20260926 01:56:10.699091 10551 raft_consensus.cc:515] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.700783 9711 ts_tablet_manager.cc:1950] T 943a990fa10948e8961c02e9999569f5 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:10.702757 10551 leader_election.cc:290] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:10.702898 8843 leader_election.cc:343] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.703330 8843 leader_election.cc:304] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:10.704071 10340 raft_consensus.cc:2749] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.703967 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.707267 9208 raft_consensus.cc:3060] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.707917 9339 tablet_bootstrap.cc:492] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:10.708384 9339 ts_tablet_manager.cc:1403] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.149s user 0.007s sys 0.003s
I20260926 01:56:10.716199 10478 raft_consensus.cc:697] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:10.717303 10478 consensus_queue.cc:237] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.731361 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:10.742437 10355 raft_consensus.cc:515] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.760038 9339 raft_consensus.cc:359] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.760908 9339 raft_consensus.cc:385] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.761725 9339 raft_consensus.cc:740] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.762759 9341 tablet_bootstrap.cc:654] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:10.764727 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:10.764894 10355 leader_election.cc:290] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:10.765376 8897 raft_consensus.cc:3060] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:10.697294 9340 raft_consensus.cc:740] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.783742 8897 raft_consensus.cc:2468] T 3f1d71caf78f4c75b144af14dce19d70 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:10.787521 10577 consensus_queue.cc:1048] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:10.789508 9003 leader_election.cc:304] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:10.792366 10577 raft_consensus.cc:2804] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:10.796456 10577 raft_consensus.cc:697] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:10.798110 10577 consensus_queue.cc:237] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.766366 9340 consensus_queue.cc:260] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.791965 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 240) took 9649 ms. Trace:
I20260926 01:56:10.817340 9341 tablet_bootstrap.cc:492] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:10.817791 9341 ts_tablet_manager.cc:1403] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.205s user 0.000s sys 0.011s
I20260926 01:56:10.825001 9341 raft_consensus.cc:359] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.825685 9341 raft_consensus.cc:385] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:10.825930 9341 raft_consensus.cc:740] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:10.826547 9341 consensus_queue.cc:260] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.830367 10340 raft_consensus.cc:493] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.833679 10340 raft_consensus.cc:515] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.837371 10340 leader_election.cc:290] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.847235 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.849190 10478 consensus_queue.cc:1048] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:10.851958 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:10.855535 8843 leader_election.cc:343] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.856324 9339 consensus_queue.cc:260] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.864912 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:10.815826 9711 rpcz_store.cc:276] 0926 01:56:01.139580 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.139692 (+ 112us) service_pool.cc:224] Handling call
0926 01:56:10.500455 (+9360763us) ts_tablet_manager.cc:1151] Deleting tablet 943a990fa10948e8961c02e9999569f5
0926 01:56:10.542342 (+ 41887us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:10.592517 (+ 50175us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:10.699320 (+106803us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:10.788414 (+ 89094us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9359630}
W20260926 01:56:10.873662 8843 leader_election.cc:343] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.874126 8843 leader_election.cc:304] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:10.875059 10340 raft_consensus.cc:493] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.883517 10340 raft_consensus.cc:515] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.885634 10340 leader_election.cc:290] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:10.886343 10340 raft_consensus.cc:2749] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.893726 9339 ts_tablet_manager.cc:1434] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.185s user 0.006s sys 0.003s
I20260926 01:56:10.894809 9339 tablet_bootstrap.cc:492] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:10.900512 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74618f7c46744d2a08cb7b4e389c9f3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:10.901994 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74618f7c46744d2a08cb7b4e389c9f3" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:10.906785 8843 leader_election.cc:343] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.895082 9340 ts_tablet_manager.cc:1434] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.198s user 0.007s sys 0.004s
I20260926 01:56:10.911615 9340 tablet_bootstrap.cc:492] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:10.916749 9341 ts_tablet_manager.cc:1434] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.099s user 0.000s sys 0.007s
W20260926 01:56:10.916697 8843 leader_election.cc:343] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.917151 8843 leader_election.cc:304] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:10.917855 10566 raft_consensus.cc:2749] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:10.918008 9341 tablet_bootstrap.cc:492] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:10.918723 9125 tablet.cc:2433] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:56:10.938470 10478 raft_consensus.cc:493] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.900712 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 239) took 9758 ms. Trace:
I20260926 01:56:10.938987 10478 raft_consensus.cc:515] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:10.939005 9711 rpcz_store.cc:276] 0926 01:56:01.139244 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.139357 (+ 113us) service_pool.cc:224] Handling call
0926 01:56:10.874893 (+9735536us) ts_tablet_manager.cc:1151] Deleting tablet 39cd9130640d4b14beef42e9f1b36b99
0926 01:56:10.888446 (+ 13553us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:10.896122 (+ 7676us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9733222}
I20260926 01:56:10.942081 10478 leader_election.cc:290] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:10.943683 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab082379974486bbea9f87f5aa64e10" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:10.945079 9003 leader_election.cc:343] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.947162 9339 tablet_bootstrap.cc:654] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:10.957686 9006 leader_election.cc:343] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.958153 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab082379974486bbea9f87f5aa64e10" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:10.958755 9006 leader_election.cc:343] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:10.959108 9006 leader_election.cc:304] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:56:10.960083 9006 leader_election.cc:343] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.961056 10577 raft_consensus.cc:2749] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:10.961586 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:10.962116 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:10.963600 10578 consensus_queue.cc:1048] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:10.970734 9268 tablet.cc:2433] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:56:10.976465 9006 leader_election.cc:343] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:10.977792 10578 consensus_queue.cc:1048] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:10.987754 10585 raft_consensus.cc:493] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:10.989037 10585 raft_consensus.cc:515] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:10.992780 10585 leader_election.cc:290] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:10.994194 8957 tablet.cc:2433] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:56:11.000306 9711 tablet_replica.cc:333] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:11.001235 9711 raft_consensus.cc:2243] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:11.001812 9711 raft_consensus.cc:2272] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:11.001675 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55a31bb9b9d4862b6bcef71df0cdaea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:11.003317 8843 leader_election.cc:343] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:11.004247 9711 ts_tablet_manager.cc:1916] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:11.008949 9006 leader_election.cc:343] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:11.010597 9152 leader_election.cc:304] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:11.017892 9006 leader_election.cc:304] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:11.014083 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55a31bb9b9d4862b6bcef71df0cdaea" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:11.031147 10578 raft_consensus.cc:2749] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:11.031561 9006 leader_election.cc:343] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:11.039688 8843 leader_election.cc:343] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:11.040150 8843 leader_election.cc:304] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:11.042449 10585 raft_consensus.cc:2749] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:11.046175 10551 raft_consensus.cc:2749] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:11.063138 9341 tablet_bootstrap.cc:654] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.060387 9340 tablet_bootstrap.cc:654] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.075249 10551 raft_consensus.cc:493] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.075794 10551 raft_consensus.cc:515] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.077909 10551 leader_election.cc:290] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:11.081019 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:11.081923 9066 raft_consensus.cc:2468] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 0.
I20260926 01:56:11.083231 9155 leader_election.cc:304] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:11.084416 10551 raft_consensus.cc:2804] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:11.085678 10551 raft_consensus.cc:493] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:11.086172 10551 raft_consensus.cc:3060] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:11.086387 9339 tablet_bootstrap.cc:492] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:11.086882 9339 ts_tablet_manager.cc:1403] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.001s
I20260926 01:56:11.089432 9339 raft_consensus.cc:359] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.090055 9339 raft_consensus.cc:385] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.134606 9339 raft_consensus.cc:740] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.135453 9339 consensus_queue.cc:260] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.137393 10551 raft_consensus.cc:515] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.159511 9339 ts_tablet_manager.cc:1434] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.072s user 0.005s sys 0.001s
I20260926 01:56:11.174533 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:11.181339 9711 ts_tablet_manager.cc:1929] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:11.181811 9711 log.cc:1199] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/cbf8818e65d74a32ba21dcb8a9a56a62
I20260926 01:56:11.182579 9711 ts_tablet_manager.cc:1950] T cbf8818e65d74a32ba21dcb8a9a56a62 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:11.211445 9339 tablet_bootstrap.cc:492] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:11.228832 9339 tablet_bootstrap.cc:654] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.242592 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 245) took 9957 ms. Trace:
I20260926 01:56:11.243031 9711 rpcz_store.cc:276] 0926 01:56:01.284705 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.285645 (+ 940us) service_pool.cc:224] Handling call
0926 01:56:10.941658 (+9656013us) ts_tablet_manager.cc:1151] Deleting tablet cbf8818e65d74a32ba21dcb8a9a56a62
0926 01:56:11.000153 (+ 58495us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.090117 (+ 89964us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.181186 (+ 91069us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.242504 (+ 61318us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9655261}
W20260926 01:56:11.225478 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 57351us
W20260926 01:56:11.257313 9152 leader_election.cc:343] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:11.273118 9339 tablet_bootstrap.cc:492] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:11.273607 9339 ts_tablet_manager.cc:1403] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.003s
I20260926 01:56:11.275094 9341 tablet_bootstrap.cc:492] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:11.275570 9341 ts_tablet_manager.cc:1403] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.358s user 0.012s sys 0.012s
I20260926 01:56:11.274963 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 247) took 9918 ms. Trace:
I20260926 01:56:11.277997 9711 rpcz_store.cc:276] 0926 01:56:01.356262 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.356372 (+ 110us) service_pool.cc:224] Handling call
0926 01:56:11.259898 (+9903526us) ts_tablet_manager.cc:1151] Deleting tablet 8d198e51b88948548cfd95dd893a26ef
0926 01:56:11.274531 (+ 14633us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.274897 (+ 366us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9902912}
I20260926 01:56:11.279233 9341 raft_consensus.cc:359] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.279892 9341 raft_consensus.cc:385] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.280114 9711 tablet_replica.cc:333] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:11.280140 9341 raft_consensus.cc:740] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.280754 9341 consensus_queue.cc:260] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.283286 9339 raft_consensus.cc:359] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.283952 9339 raft_consensus.cc:385] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.284199 9339 raft_consensus.cc:740] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.284886 9339 consensus_queue.cc:260] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.293444 9341 ts_tablet_manager.cc:1434] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260926 01:56:11.293809 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:11.294447 9066 raft_consensus.cc:3060] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:11.300220 9341 tablet_bootstrap.cc:492] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:11.304466 9066 raft_consensus.cc:2468] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:11.305936 9155 leader_election.cc:304] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:11.308415 9339 ts_tablet_manager.cc:1434] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260926 01:56:11.321208 9711 raft_consensus.cc:2243] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:11.321841 9711 raft_consensus.cc:2272] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:11.324031 9711 ts_tablet_manager.cc:1916] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:11.331673 9339 tablet_bootstrap.cc:492] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:11.332798 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:11.306404 9341 tablet_bootstrap.cc:654] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.306860 10546 raft_consensus.cc:2804] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:11.347002 10546 raft_consensus.cc:697] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:11.347878 10546 consensus_queue.cc:237] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.334110 10551 leader_election.cc:290] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:11.357112 9340 tablet_bootstrap.cc:492] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:11.357615 9340 ts_tablet_manager.cc:1403] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.447s user 0.019s sys 0.011s
I20260926 01:56:11.360582 9340 raft_consensus.cc:359] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.361281 9340 raft_consensus.cc:385] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.361543 9340 raft_consensus.cc:740] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:56:11.363770 9152 leader_election.cc:343] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:11.387648 9339 tablet_bootstrap.cc:654] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.362207 9340 consensus_queue.cc:260] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.396659 9340 ts_tablet_manager.cc:1434] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20260926 01:56:11.409510 10546 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:11.410180 10546 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:11.416199 9155 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:11.416918 10546 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:11.418821 9339 tablet_bootstrap.cc:492] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:11.419346 9339 ts_tablet_manager.cc:1403] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.000s
I20260926 01:56:11.428012 9341 tablet_bootstrap.cc:492] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:11.428524 9341 ts_tablet_manager.cc:1403] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.134s user 0.011s sys 0.000s
I20260926 01:56:11.442750 9341 raft_consensus.cc:359] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.443473 9341 raft_consensus.cc:385] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.443732 9341 raft_consensus.cc:740] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.444397 9341 consensus_queue.cc:260] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.446499 9339 raft_consensus.cc:359] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.447225 9339 raft_consensus.cc:385] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.450676 9339 raft_consensus.cc:740] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.451597 9339 consensus_queue.cc:260] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.453724 9339 ts_tablet_manager.cc:1434] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260926 01:56:11.454733 9339 tablet_bootstrap.cc:492] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:11.429252 9711 ts_tablet_manager.cc:1929] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:11.458389 9711 log.cc:1199] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/1f6d9fedc7b241b59b25d591c7dc2fea
I20260926 01:56:11.467443 9711 ts_tablet_manager.cc:1950] T 1f6d9fedc7b241b59b25d591c7dc2fea P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:11.472240 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 246) took 10116 ms. Trace:
I20260926 01:56:11.472604 9711 rpcz_store.cc:276] 0926 01:56:01.355906 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:01.397472 (+ 41566us) service_pool.cc:224] Handling call
0926 01:56:11.279926 (+9882454us) ts_tablet_manager.cc:1151] Deleting tablet 1f6d9fedc7b241b59b25d591c7dc2fea
0926 01:56:11.279995 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.380208 (+100213us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.429094 (+ 48886us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.472161 (+ 43067us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40238,"tablet-delete.queue_time_us":9881801}
I20260926 01:56:11.478026 9711 tablet_replica.cc:333] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:11.481740 9341 ts_tablet_manager.cc:1434] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.053s user 0.009s sys 0.000s
I20260926 01:56:11.485639 9711 raft_consensus.cc:2243] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:11.486649 9711 raft_consensus.cc:2272] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:11.493506 9341 tablet_bootstrap.cc:492] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:11.387152 10566 consensus_queue.cc:1048] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.067s
I20260926 01:56:11.496754 9340 tablet_bootstrap.cc:492] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:11.509507 9711 ts_tablet_manager.cc:1916] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 01:56:11.533866 9152 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:11.536468 9339 tablet_bootstrap.cc:654] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.534336 9152 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:11.558888 10546 raft_consensus.cc:2749] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:11.537061 9341 tablet_bootstrap.cc:654] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.581424 10546 raft_consensus.cc:493] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.582111 10546 raft_consensus.cc:515] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:11.585549 10546 leader_election.cc:290] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:11.585361 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:11.607066 10546 raft_consensus.cc:493] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.607896 10546 raft_consensus.cc:515] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.610287 9217 raft_consensus.cc:3060] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:11.620990 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cbf646569e6497dac90e308f0681372" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:11.623845 10546 leader_election.cc:290] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:11.633044 9341 tablet_bootstrap.cc:492] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:11.633522 9341 ts_tablet_manager.cc:1403] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.140s user 0.012s sys 0.001s
I20260926 01:56:11.636847 10589 consensus_queue.cc:1048] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:11.642714 9341 raft_consensus.cc:359] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.643428 9341 raft_consensus.cc:385] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.643713 9341 raft_consensus.cc:740] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.644471 9341 consensus_queue.cc:260] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.647113 9341 ts_tablet_manager.cc:1434] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260926 01:56:11.678254 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:11.682165 9208 raft_consensus.cc:3060] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:11.683458 9341 tablet_bootstrap.cc:492] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:11.690300 10589 raft_consensus.cc:493] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.690891 10589 raft_consensus.cc:515] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.694798 10589 leader_election.cc:290] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:11.696309 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:11.697202 9066 raft_consensus.cc:2393] T 5da0c41e2e19409d938f8f8ec29cb91a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate bc00feec3f564608851f0112e97ab532 in this term.
I20260926 01:56:11.699532 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da0c41e2e19409d938f8f8ec29cb91a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:11.701228 8843 leader_election.cc:304] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:11.702589 10589 raft_consensus.cc:3060] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:11.725581 9341 tablet_bootstrap.cc:654] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.736294 9339 tablet_bootstrap.cc:492] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:11.736833 9339 ts_tablet_manager.cc:1403] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.282s user 0.009s sys 0.004s
I20260926 01:56:11.741979 9339 raft_consensus.cc:359] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.742766 9339 raft_consensus.cc:385] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.742961 9340 tablet_bootstrap.cc:654] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.743041 9339 raft_consensus.cc:740] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.744853 9339 consensus_queue.cc:260] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.746381 9339 ts_tablet_manager.cc:1434] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260926 01:56:11.726254 9711 ts_tablet_manager.cc:1929] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:11.762343 9711 log.cc:1199] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/bcb000f9234d4382ad019f63015b92e7
I20260926 01:56:11.763248 9711 ts_tablet_manager.cc:1950] T bcb000f9234d4382ad019f63015b92e7 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:11.777822 10608 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:11.778553 10608 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:11.784258 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:11.795831 9339 tablet_bootstrap.cc:492] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:11.812763 10608 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:11.818914 9339 tablet_bootstrap.cc:654] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.818885 10496 raft_consensus.cc:493] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.819422 10496 raft_consensus.cc:515] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.821409 10496 leader_election.cc:290] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:11.825261 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:11.827723 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 249) took 9249 ms. Trace:
I20260926 01:56:11.828078 9711 rpcz_store.cc:276] 0926 01:56:02.580534 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:02.580671 (+ 137us) service_pool.cc:224] Handling call
0926 01:56:11.477838 (+8897167us) ts_tablet_manager.cc:1151] Deleting tablet bcb000f9234d4382ad019f63015b92e7
0926 01:56:11.477897 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.561028 (+ 83131us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.726100 (+165072us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.827647 (+101547us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8894368}
I20260926 01:56:11.829460 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 250) took 8451 ms. Trace:
I20260926 01:56:11.829780 9711 rpcz_store.cc:276] 0926 01:56:03.378407 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:03.378544 (+ 137us) service_pool.cc:224] Handling call
0926 01:56:11.829038 (+8450494us) ts_tablet_manager.cc:1151] Deleting tablet b612e8955d5f42dd8658320031fe36ca
0926 01:56:11.829105 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.829406 (+ 301us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8449830}
I20260926 01:56:11.830991 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 251) took 7763 ms. Trace:
I20260926 01:56:11.831285 9711 rpcz_store.cc:276] 0926 01:56:04.068072 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:04.068196 (+ 124us) service_pool.cc:224] Handling call
0926 01:56:11.830616 (+7762420us) ts_tablet_manager.cc:1151] Deleting tablet 00b4ea0a3f8a40eb90fd545ca6df7a8e
0926 01:56:11.830676 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.830930 (+ 254us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7761797}
I20260926 01:56:11.831735 10577 consensus_queue.cc:1048] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:11.832505 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 265) took 4683 ms. Trace:
I20260926 01:56:11.832887 9711 rpcz_store.cc:276] 0926 01:56:07.149548 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.151244 (+ 1696us) service_pool.cc:224] Handling call
0926 01:56:11.832275 (+4681031us) ts_tablet_manager.cc:1151] Deleting tablet c36ea09cf15647c0a344c9ba9b289d61
0926 01:56:11.832316 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.832453 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4548960}
I20260926 01:56:11.833969 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 264) took 4684 ms. Trace:
I20260926 01:56:11.834273 9711 rpcz_store.cc:276] 0926 01:56:07.149209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.151785 (+ 2576us) service_pool.cc:224] Handling call
0926 01:56:11.833731 (+4681946us) ts_tablet_manager.cc:1151] Deleting tablet c27f1dd7bb0449a6a3fa371a8636e09b
0926 01:56:11.833773 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.833914 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4550197}
I20260926 01:56:11.835251 9711 tablet_replica.cc:333] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:11.835991 9711 raft_consensus.cc:2243] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:11.836794 9711 raft_consensus.cc:2272] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:11.852548 9340 tablet_bootstrap.cc:492] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:11.857781 10577 consensus_queue.cc:1048] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:11.859766 10577 raft_consensus.cc:493] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.860209 10577 raft_consensus.cc:515] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.863183 10577 leader_election.cc:290] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:11.863976 9340 ts_tablet_manager.cc:1403] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.389s user 0.013s sys 0.004s
I20260926 01:56:11.873720 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55a31bb9b9d4862b6bcef71df0cdaea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:11.874478 8899 raft_consensus.cc:2468] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:11.876157 9340 raft_consensus.cc:359] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:11.877210 9340 raft_consensus.cc:385] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.877764 9340 raft_consensus.cc:740] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.878551 9340 consensus_queue.cc:260] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:11.880724 9340 ts_tablet_manager.cc:1434] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260926 01:56:11.881733 9340 tablet_bootstrap.cc:492] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:11.827304 9152 leader_election.cc:304] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:11.884063 10496 raft_consensus.cc:2749] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:11.884122 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 56952.4us
I20260926 01:56:11.859953 10589 raft_consensus.cc:2749] T 5da0c41e2e19409d938f8f8ec29cb91a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:11.887056 9220 raft_consensus.cc:3060] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:11.894131 9341 tablet_bootstrap.cc:492] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:11.894635 9341 ts_tablet_manager.cc:1403] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.211s user 0.012s sys 0.000s
I20260926 01:56:11.896384 9711 ts_tablet_manager.cc:1916] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:11.897320 9341 raft_consensus.cc:359] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.897966 9341 raft_consensus.cc:385] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.898339 9339 tablet_bootstrap.cc:492] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:11.898795 9339 ts_tablet_manager.cc:1403] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.007s
I20260926 01:56:11.901382 9339 raft_consensus.cc:359] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.901997 9339 raft_consensus.cc:385] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.903249 9340 tablet_bootstrap.cc:654] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:11.928974 10577 raft_consensus.cc:493] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.929544 10577 raft_consensus.cc:515] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.931671 10577 leader_election.cc:290] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:11.936394 9340 tablet_bootstrap.cc:492] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:11.936916 9340 ts_tablet_manager.cc:1403] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.005s
I20260926 01:56:11.939922 9340 raft_consensus.cc:359] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.902256 9339 raft_consensus.cc:740] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.942924 9340 raft_consensus.cc:385] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:11.943388 9340 raft_consensus.cc:740] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:11.943296 9339 consensus_queue.cc:260] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.944576 9340 consensus_queue.cc:260] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.945518 9339 ts_tablet_manager.cc:1434] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20260926 01:56:11.947515 9340 ts_tablet_manager.cc:1434] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260926 01:56:11.949131 10577 raft_consensus.cc:493] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:11.949599 10577 raft_consensus.cc:515] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:11.951714 10577 leader_election.cc:290] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:11.959488 9711 ts_tablet_manager.cc:1929] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:11.959944 9711 log.cc:1199] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:11.960690 9711 ts_tablet_manager.cc:1950] T 1743e863cdd84f3f8ca413fa81cd1914 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:11.965266 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55a31bb9b9d4862b6bcef71df0cdaea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:11.965483 9210 raft_consensus.cc:3060] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:11.967347 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:11.970548 9214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:11.985524 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 266) took 4835 ms. Trace:
I20260926 01:56:11.986032 9711 rpcz_store.cc:276] 0926 01:56:07.149959 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.152083 (+ 2124us) service_pool.cc:224] Handling call
0926 01:56:11.835117 (+4683034us) ts_tablet_manager.cc:1151] Deleting tablet 1743e863cdd84f3f8ca413fa81cd1914
0926 01:56:11.835153 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.922403 (+ 87250us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.959333 (+ 36930us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:11.985434 (+ 26101us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4551410}
I20260926 01:56:11.988723 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 268) took 4835 ms. Trace:
I20260926 01:56:11.998035 9711 rpcz_store.cc:276] 0926 01:56:07.152846 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.153273 (+ 427us) service_pool.cc:224] Handling call
0926 01:56:11.987518 (+4834245us) ts_tablet_manager.cc:1151] Deleting tablet 70ef5265eb6244aa96c8195aa30cdfb0
0926 01:56:11.987573 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.988656 (+ 1083us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4703612}
I20260926 01:56:11.999681 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 269) took 4835 ms. Trace:
I20260926 01:56:12.000123 9711 rpcz_store.cc:276] 0926 01:56:07.164435 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.164618 (+ 183us) service_pool.cc:224] Handling call
0926 01:56:11.999339 (+4834721us) ts_tablet_manager.cc:1151] Deleting tablet 3bf2fa5fbe4146b5adb27ff09de878d3
0926 01:56:11.999392 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:11.999618 (+ 226us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4715281}
I20260926 01:56:12.001395 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 270) took 4836 ms. Trace:
I20260926 01:56:12.008141 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f1d71caf78f4c75b144af14dce19d70" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:12.011006 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cbf646569e6497dac90e308f0681372" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:11.898241 9341 raft_consensus.cc:740] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.016515 9341 consensus_queue.cc:260] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.017529 9208 raft_consensus.cc:2468] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:12.022195 9341 ts_tablet_manager.cc:1434] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.127s user 0.005s sys 0.000s
I20260926 01:56:12.023437 9341 tablet_bootstrap.cc:492] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:12.036659 9340 tablet_bootstrap.cc:492] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:12.015347 9003 leader_election.cc:304] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:12.040446 10577 raft_consensus.cc:2804] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:12.040812 10577 raft_consensus.cc:493] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:12.041126 10577 raft_consensus.cc:3060] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.016604 9339 tablet_bootstrap.cc:492] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:12.047542 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:12.048238 8905 raft_consensus.cc:2468] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:12.048293 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:12.049106 8906 raft_consensus.cc:2468] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:12.051860 9340 tablet_bootstrap.cc:654] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.039929 9711 rpcz_store.cc:276] 0926 01:56:07.164870 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.165055 (+ 185us) service_pool.cc:224] Handling call
0926 01:56:12.001136 (+4836081us) ts_tablet_manager.cc:1151] Deleting tablet 17b8b0d54f5645e5963f1de217f94c84
0926 01:56:12.001180 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:12.001343 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4716874}
I20260926 01:56:12.064487 9711 tablet_replica.cc:333] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:12.074836 10123 mvcc.cc:204] Tried to move back new op lower bound from 7333428252400373760 to 7333428182869704704. Current Snapshot: MvccSnapshot[applied={T|T < 7333428252400373760 or (T in {7333428252400373760})}]
I20260926 01:56:12.094630 9711 raft_consensus.cc:2243] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:12.095276 9711 raft_consensus.cc:2272] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:12.097483 9711 ts_tablet_manager.cc:1916] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:12.101717 10355 consensus_queue.cc:1048] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:12.075294 10577 raft_consensus.cc:515] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.113726 10577 leader_election.cc:290] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:12.125427 10625 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:12.125576 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55a31bb9b9d4862b6bcef71df0cdaea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:12.126156 8897 raft_consensus.cc:3060] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.125926 10625 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
W20260926 01:56:12.124835 9006 leader_election.cc:343] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.134073 9340 tablet_bootstrap.cc:492] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:12.134662 9340 ts_tablet_manager.cc:1403] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.098s user 0.013s sys 0.009s
I20260926 01:56:12.137809 8897 raft_consensus.cc:2468] T e55a31bb9b9d4862b6bcef71df0cdaea P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:12.144876 9003 leader_election.cc:304] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:12.145200 10551 consensus_queue.cc:1048] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:12.145990 9340 raft_consensus.cc:359] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.146827 9340 raft_consensus.cc:385] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:12.147074 9340 raft_consensus.cc:740] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.147735 9340 consensus_queue.cc:260] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:12.152753 9006 leader_election.cc:343] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.155562 9340 ts_tablet_manager.cc:1434] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260926 01:56:12.156564 9340 tablet_bootstrap.cc:492] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:12.159075 10355 raft_consensus.cc:2804] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:12.159482 10355 raft_consensus.cc:493] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:12.159785 10355 raft_consensus.cc:3060] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.163108 9003 leader_election.cc:304] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:12.163903 10478 raft_consensus.cc:2804] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:12.164258 10478 raft_consensus.cc:493] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:12.164558 10478 raft_consensus.cc:3060] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.165120 9003 leader_election.cc:304] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:12.167433 10621 raft_consensus.cc:2804] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:12.167847 10621 raft_consensus.cc:697] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:12.168630 10621 consensus_queue.cc:237] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.172642 10478 raft_consensus.cc:515] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.176826 9339 tablet_bootstrap.cc:654] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.183166 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:12.187011 10478 leader_election.cc:290] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:12.188482 9340 tablet_bootstrap.cc:654] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.195052 9341 tablet_bootstrap.cc:654] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.201310 10478 consensus_queue.cc:1048] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:12.208715 9340 tablet_bootstrap.cc:492] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:12.209183 9340 ts_tablet_manager.cc:1403] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.004s
I20260926 01:56:12.212203 9711 ts_tablet_manager.cc:1929] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:12.211954 9340 raft_consensus.cc:359] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:12.212466 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet e8e0e6ede1ce4487b08bfd5743e3656a consensus state has no leader
I20260926 01:56:12.212612 9711 log.cc:1199] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/92896db04df346bf91f732b1ce89cd23
I20260926 01:56:12.212637 9340 raft_consensus.cc:385] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:12.212874 9340 raft_consensus.cc:740] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.213379 9711 ts_tablet_manager.cc:1950] T 92896db04df346bf91f732b1ce89cd23 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:12.213483 9340 consensus_queue.cc:260] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.215077 10355 raft_consensus.cc:515] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.215516 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 271) took 5022 ms. Trace:
I20260926 01:56:12.215823 9711 rpcz_store.cc:276] 0926 01:56:07.193530 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.194130 (+ 600us) service_pool.cc:224] Handling call
0926 01:56:12.064297 (+4870167us) ts_tablet_manager.cc:1151] Deleting tablet 92896db04df346bf91f732b1ce89cd23
0926 01:56:12.064349 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:12.158112 (+ 93763us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:12.212043 (+ 53931us) tablet_metadata.cc:665] Metadata flushed
0926 01:56:12.215453 (+ 3410us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29243,"tablet-delete.queue_time_us":4779635}
I20260926 01:56:12.217088 10355 leader_election.cc:290] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:12.217146 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 275) took 4767 ms. Trace:
I20260926 01:56:12.219846 9340 ts_tablet_manager.cc:1434] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260926 01:56:12.222373 10625 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:12.218367 9711 rpcz_store.cc:276] 0926 01:56:07.450025 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.450160 (+ 135us) service_pool.cc:224] Handling call
0926 01:56:12.216875 (+4766715us) ts_tablet_manager.cc:1151] Deleting tablet 16f49a5a10ea4f7a821d89f18e3af560
0926 01:56:12.216919 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:12.217096 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4711156}
I20260926 01:56:12.233070 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 277) took 4751 ms. Trace:
I20260926 01:56:12.240279 9711 rpcz_store.cc:276] 0926 01:56:07.481980 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.486635 (+ 4655us) service_pool.cc:224] Handling call
0926 01:56:12.232756 (+4746121us) ts_tablet_manager.cc:1151] Deleting tablet 6d56f3a5dfed4ac78f1da92e22db558a
0926 01:56:12.232807 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:12.233009 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":503,"tablet-delete.queue_time_us":4726351}
I20260926 01:56:12.241595 9711 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51458 (request call id 278) took 4735 ms. Trace:
I20260926 01:56:12.241938 9711 rpcz_store.cc:276] 0926 01:56:07.506648 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:07.506788 (+ 140us) service_pool.cc:224] Handling call
0926 01:56:12.241310 (+4734522us) ts_tablet_manager.cc:1151] Deleting tablet 2e12827ad93c47cf9013e238215ceb7a
0926 01:56:12.241358 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0926 01:56:12.241537 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4729542}
W20260926 01:56:12.274942 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 109877us
I20260926 01:56:12.277849 10589 raft_consensus.cc:493] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:12.278354 10589 raft_consensus.cc:515] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.280354 10589 leader_election.cc:290] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:12.282164 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:12.283573 8843 leader_election.cc:343] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.286810 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:12.287453 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:12.288046 8897 raft_consensus.cc:3060] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.290246 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:12.290831 8905 raft_consensus.cc:3060] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:12.291927 9910 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.509s (SENT) [suppressed 2 similar messages]
I20260926 01:56:12.299654 8897 raft_consensus.cc:2468] T 3c26f557ac274973824e6f9a4532de74 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:12.307915 8905 raft_consensus.cc:2468] T 6f7419abcc2c482e97df70f03e047450 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:12.356523 9341 tablet_bootstrap.cc:492] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:12.357045 9341 ts_tablet_manager.cc:1403] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.334s user 0.011s sys 0.000s
W20260926 01:56:12.367969 8843 leader_election.cc:343] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.368465 8843 leader_election.cc:304] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:12.369534 10340 raft_consensus.cc:2749] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:12.388458 10638 tablet.cc:1751] T c9f54f648a55404fb6622b35d84368ee P d20f26c9295641ecbf77d290ca206bb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:12.411403 9341 raft_consensus.cc:359] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.412277 9341 raft_consensus.cc:385] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:12.412577 9341 raft_consensus.cc:740] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.413277 9341 consensus_queue.cc:260] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.420719 10551 raft_consensus.cc:493] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:12.421180 10551 raft_consensus.cc:515] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.423202 10551 leader_election.cc:290] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:12.424748 10637 tablet.cc:1751] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:12.425601 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e55a31bb9b9d4862b6bcef71df0cdaea" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:12.427573 9341 ts_tablet_manager.cc:1434] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.070s user 0.008s sys 0.000s
I20260926 01:56:12.428643 9341 tablet_bootstrap.cc:492] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:12.428961 9214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:12.429183 9003 leader_election.cc:304] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:12.430235 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:12.432307 9210 raft_consensus.cc:3060] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.439248 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:12.440709 9152 leader_election.cc:304] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:12.443496 9210 raft_consensus.cc:2468] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:12.452633 9339 tablet_bootstrap.cc:492] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:12.453109 9339 ts_tablet_manager.cc:1403] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.437s user 0.003s sys 0.016s
I20260926 01:56:12.459125 9340 tablet_bootstrap.cc:492] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:12.464946 9340 tablet_bootstrap.cc:654] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.466784 9341 tablet_bootstrap.cc:654] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.467761 10496 raft_consensus.cc:2749] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:12.471076 10630 raft_consensus.cc:2804] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:12.471619 10630 raft_consensus.cc:697] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:12.472470 10630 consensus_queue.cc:237] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.477990 9003 leader_election.cc:304] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:12.478900 10626 raft_consensus.cc:2804] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:12.479346 10626 raft_consensus.cc:697] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:12.480159 10626 consensus_queue.cc:237] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.483887 9339 raft_consensus.cc:359] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.484583 9339 raft_consensus.cc:385] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:12.484838 9339 raft_consensus.cc:740] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.485498 9339 consensus_queue.cc:260] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.534160 9339 ts_tablet_manager.cc:1434] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.081s user 0.007s sys 0.000s
I20260926 01:56:12.442296 9152 leader_election.cc:304] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:12.538053 10496 raft_consensus.cc:2749] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:12.538192 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 95978.3us
I20260926 01:56:12.539520 10546 consensus_queue.cc:1048] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:12.542672 10355 raft_consensus.cc:493] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:12.543169 10355 raft_consensus.cc:515] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.545521 10355 leader_election.cc:290] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:12.566356 10589 raft_consensus.cc:493] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:12.566923 10589 raft_consensus.cc:515] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.568991 10589 leader_election.cc:290] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:12.570673 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030f4d683e404fbd8d2e6863e18f71c0" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:12.571400 9220 raft_consensus.cc:2468] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:12.573547 8843 leader_election.cc:304] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:12.574401 10589 raft_consensus.cc:2804] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:12.574767 10589 raft_consensus.cc:493] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:12.575093 10589 raft_consensus.cc:3060] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.582472 10589 raft_consensus.cc:515] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.585361 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030f4d683e404fbd8d2e6863e18f71c0" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:12.586010 9220 raft_consensus.cc:3060] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.590723 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:12.594548 9220 raft_consensus.cc:2468] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:12.596174 8843 leader_election.cc:304] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:12.596994 10340 raft_consensus.cc:2804] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:12.597466 10340 raft_consensus.cc:697] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:12.598320 10340 consensus_queue.cc:237] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.605368 9339 tablet_bootstrap.cc:492] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:12.607851 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 68274.1us
I20260926 01:56:12.609380 9341 tablet_bootstrap.cc:492] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:12.610035 9341 ts_tablet_manager.cc:1403] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.182s user 0.021s sys 0.002s
I20260926 01:56:12.635063 9341 raft_consensus.cc:359] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:12.636006 9341 raft_consensus.cc:385] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:12.636346 9341 raft_consensus.cc:740] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.637042 9341 consensus_queue.cc:260] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:12.654273 9341 ts_tablet_manager.cc:1434] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.044s user 0.001s sys 0.005s
I20260926 01:56:12.660436 9340 tablet_bootstrap.cc:492] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:12.661165 9340 ts_tablet_manager.cc:1403] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.202s user 0.028s sys 0.012s
I20260926 01:56:12.664168 9340 raft_consensus.cc:359] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.664839 9340 raft_consensus.cc:385] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:12.665086 9340 raft_consensus.cc:740] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.665721 9340 consensus_queue.cc:260] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.671566 10589 leader_election.cc:290] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:12.677479 9006 leader_election.cc:343] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.677907 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eef6a48e87e4c61a23058942a82bccc" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:12.678993 9006 leader_election.cc:343] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.698988 10618 raft_consensus.cc:493] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:12.699316 9340 ts_tablet_manager.cc:1434] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20260926 01:56:12.699560 10618 raft_consensus.cc:515] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:12.701087 9340 tablet_bootstrap.cc:492] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:12.704046 10618 leader_election.cc:290] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:12.704334 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab082379974486bbea9f87f5aa64e10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:12.704981 9220 raft_consensus.cc:2468] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:12.706122 8843 leader_election.cc:304] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:12.709868 10340 raft_consensus.cc:2804] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:12.710258 10340 raft_consensus.cc:493] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:12.710664 10340 raft_consensus.cc:3060] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.719844 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:12.732086 9341 tablet_bootstrap.cc:492] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:12.737805 9340 tablet_bootstrap.cc:654] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.769281 9339 tablet_bootstrap.cc:654] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:12.740818 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 66230us
W20260926 01:56:12.779633 9003 leader_election.cc:343] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.802752 10340 raft_consensus.cc:515] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:12.805859 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 65692.7us
I20260926 01:56:12.806630 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab082379974486bbea9f87f5aa64e10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:12.807231 9217 raft_consensus.cc:3060] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.811223 9152 leader_election.cc:304] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:12.815996 9217 raft_consensus.cc:2468] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:12.817250 8843 leader_election.cc:304] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
W20260926 01:56:12.832787 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.509s (SENT)
I20260926 01:56:12.833274 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:12.834848 10655 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:12.876081 10635 raft_consensus.cc:2749] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:12.883358 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 72295.5us
I20260926 01:56:12.892270 10589 raft_consensus.cc:2804] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:12.901444 10589 raft_consensus.cc:697] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:12.903615 9341 tablet_bootstrap.cc:654] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:12.908030 10589 consensus_queue.cc:237] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:12.912869 10621 raft_consensus.cc:493] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:12.913383 10621 raft_consensus.cc:515] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.916801 10621 leader_election.cc:290] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:12.918678 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030f4d683e404fbd8d2e6863e18f71c0" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:12.919477 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab082379974486bbea9f87f5aa64e10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:12.920091 9069 raft_consensus.cc:2468] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:12.920773 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030f4d683e404fbd8d2e6863e18f71c0" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
W20260926 01:56:12.926230 9006 leader_election.cc:343] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:12.926746 9006 leader_election.cc:304] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:12.927985 10578 raft_consensus.cc:2749] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:12.928269 9339 tablet_bootstrap.cc:492] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:12.928702 9339 ts_tablet_manager.cc:1403] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.324s user 0.010s sys 0.002s
I20260926 01:56:12.933957 9339 raft_consensus.cc:359] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.935125 9339 raft_consensus.cc:385] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:12.935385 9339 raft_consensus.cc:740] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:12.935994 9339 consensus_queue.cc:260] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.938979 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab082379974486bbea9f87f5aa64e10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:12.939541 9068 raft_consensus.cc:3060] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:12.947949 9068 raft_consensus.cc:2468] T 6ab082379974486bbea9f87f5aa64e10 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:12.982908 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e140cf11b8bb49f9a94c0158a2afca4d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:12.987564 10578 raft_consensus.cc:493] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:12.988128 10578 raft_consensus.cc:515] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:12.991330 10578 leader_election.cc:290] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:12.992522 9341 tablet_bootstrap.cc:492] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:12.993003 9341 ts_tablet_manager.cc:1403] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.338s user 0.011s sys 0.001s
I20260926 01:56:12.987087 9339 ts_tablet_manager.cc:1434] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.058s user 0.008s sys 0.000s
W20260926 01:56:13.010756 9006 leader_election.cc:343] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:13.018476 8843 leader_election.cc:343] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:13.024435 8843 leader_election.cc:343] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.028304 8895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74618f7c46744d2a08cb7b4e389c9f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:13.029592 8895 raft_consensus.cc:2468] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:13.037853 8898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e140cf11b8bb49f9a94c0158a2afca4d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:12.892280 10340 leader_election.cc:290] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:13.053414 10340 raft_consensus.cc:493] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:13.054363 10340 raft_consensus.cc:515] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.057660 10340 leader_election.cc:290] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:13.066973 9003 leader_election.cc:304] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:13.067848 10630 raft_consensus.cc:2804] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:13.068178 10630 raft_consensus.cc:493] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:13.068521 10630 raft_consensus.cc:3060] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:13.074116 9341 raft_consensus.cc:359] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.074831 9341 raft_consensus.cc:385] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.075089 9341 raft_consensus.cc:740] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.080889 10630 raft_consensus.cc:515] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:13.086216 9003 leader_election.cc:343] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.086856 9003 leader_election.cc:304] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:13.087960 10578 raft_consensus.cc:2749] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:13.092315 9339 tablet_bootstrap.cc:492] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:13.095961 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74618f7c46744d2a08cb7b4e389c9f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.114116 9340 tablet_bootstrap.cc:492] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:13.115070 9340 ts_tablet_manager.cc:1403] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.414s user 0.023s sys 0.027s
I20260926 01:56:13.120396 10630 leader_election.cc:290] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:13.130218 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74618f7c46744d2a08cb7b4e389c9f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:13.130980 8901 raft_consensus.cc:3060] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:13.122659 9340 raft_consensus.cc:359] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.133157 9340 raft_consensus.cc:385] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.133497 9340 raft_consensus.cc:740] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.134202 9340 consensus_queue.cc:260] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.075779 9341 consensus_queue.cc:260] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.139216 9220 raft_consensus.cc:2468] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:13.141543 10578 consensus_queue.cc:1048] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:13.154374 9340 ts_tablet_manager.cc:1434] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20260926 01:56:13.175184 9339 tablet_bootstrap.cc:654] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:13.188526 9341 ts_tablet_manager.cc:1434] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.195s user 0.008s sys 0.000s
I20260926 01:56:13.189754 9341 tablet_bootstrap.cc:492] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:13.204087 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74618f7c46744d2a08cb7b4e389c9f3" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:13.204763 9220 raft_consensus.cc:3060] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:13.220808 10652 raft_consensus.cc:493] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:13.221434 10652 raft_consensus.cc:515] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.223608 10652 leader_election.cc:290] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:13.225987 9340 tablet_bootstrap.cc:492] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:13.220898 10666 tablet.cc:1751] T c9f54f648a55404fb6622b35d84368ee P 49bd320bf73f4252804ec867ec077510: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260926 01:56:13.216753 9339 tablet_bootstrap.cc:492] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:13.244247 10630 consensus_queue.cc:1048] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:13.245832 9220 raft_consensus.cc:2468] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
W20260926 01:56:13.248204 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 222078us
I20260926 01:56:13.259608 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8711788a85f94c599224123f6b8cc814" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.263376 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8711788a85f94c599224123f6b8cc814" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:13.272625 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e140cf11b8bb49f9a94c0158a2afca4d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.226683 8901 raft_consensus.cc:2468] T f74618f7c46744d2a08cb7b4e389c9f3 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:13.276787 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e140cf11b8bb49f9a94c0158a2afca4d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:13.277385 9068 raft_consensus.cc:2468] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:13.279589 9006 leader_election.cc:304] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:13.280642 10621 raft_consensus.cc:2804] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:13.281095 10621 raft_consensus.cc:697] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:13.281915 10621 consensus_queue.cc:237] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:13.301688 8843 leader_election.cc:343] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.307546 9340 tablet_bootstrap.cc:654] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:13.329403 9006 consensus_peers.cc:597] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:13.243983 9339 ts_tablet_manager.cc:1403] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.152s user 0.009s sys 0.003s
I20260926 01:56:13.337494 9339 raft_consensus.cc:359] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:13.338201 9339 raft_consensus.cc:385] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.351358 8843 leader_election.cc:304] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:13.354717 10630 raft_consensus.cc:493] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:13.356264 10340 raft_consensus.cc:2804] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:13.355259 10630 raft_consensus.cc:515] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.356578 10340 raft_consensus.cc:493] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:13.357218 10340 raft_consensus.cc:3060] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:13.358564 10630 leader_election.cc:290] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:13.365430 10340 raft_consensus.cc:515] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:13.366942 8843 leader_election.cc:343] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.369330 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.372529 10635 raft_consensus.cc:493] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:13.373075 10635 raft_consensus.cc:515] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:13.375293 10635 leader_election.cc:290] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:13.376091 9341 tablet_bootstrap.cc:654] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:13.377295 9155 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
W20260926 01:56:13.378608 9006 leader_election.cc:343] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.379633 9339 raft_consensus.cc:740] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.380393 9339 consensus_queue.cc:260] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:13.382329 9340 tablet_bootstrap.cc:492] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:13.382628 9339 ts_tablet_manager.cc:1434] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260926 01:56:13.382822 9340 ts_tablet_manager.cc:1403] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.157s user 0.012s sys 0.000s
I20260926 01:56:13.383637 9339 tablet_bootstrap.cc:492] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:13.385373 9340 raft_consensus.cc:359] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.385998 9340 raft_consensus.cc:385] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.386256 9340 raft_consensus.cc:740] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.368877 8843 leader_election.cc:304] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:13.390496 10662 raft_consensus.cc:2749] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:13.391820 9340 consensus_queue.cc:260] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:13.387243 9152 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
I20260926 01:56:13.394840 9152 leader_election.cc:304] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:13.401058 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:13.401732 8897 raft_consensus.cc:2468] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:13.404289 9340 ts_tablet_manager.cc:1434] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260926 01:56:13.405460 9340 tablet_bootstrap.cc:492] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:13.414968 8843 leader_election.cc:343] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:13.419265 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.612s (SENT)
I20260926 01:56:13.419741 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:13.430945 10635 raft_consensus.cc:2749] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:13.432067 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 44884.4us
W20260926 01:56:13.438324 9006 consensus_peers.cc:597] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:13.439978 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e140cf11b8bb49f9a94c0158a2afca4d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:13.443502 10340 leader_election.cc:290] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:13.450137 10662 raft_consensus.cc:493] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:13.450675 10662 raft_consensus.cc:515] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.452859 10662 leader_election.cc:290] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:13.463425 9340 tablet_bootstrap.cc:654] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:13.464592 10578 consensus_queue.cc:1048] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:13.465193 10621 consensus_queue.cc:1048] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:13.467195 10652 raft_consensus.cc:493] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:13.467440 9003 leader_election.cc:304] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:13.467710 10652 raft_consensus.cc:515] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.468199 10621 raft_consensus.cc:2804] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:13.468554 10621 raft_consensus.cc:493] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:13.468871 10621 raft_consensus.cc:3060] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:13.469626 10652 leader_election.cc:290] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:13.470101 10662 consensus_queue.cc:1048] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:13.472216 8843 leader_election.cc:343] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.479658 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.491799 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.495308 10675 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260926 01:56:13.498188 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 78978.5us
W20260926 01:56:13.506196 8843 leader_election.cc:343] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:13.516834 8843 leader_election.cc:343] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.493922 10621 raft_consensus.cc:515] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.523085 10621 leader_election.cc:290] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:13.528021 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:13.530198 9006 leader_election.cc:343] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.525285 9340 tablet_bootstrap.cc:492] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:13.539736 9340 ts_tablet_manager.cc:1403] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.135s user 0.025s sys 0.003s
I20260926 01:56:13.542341 9340 raft_consensus.cc:359] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.543001 9340 raft_consensus.cc:385] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.543239 9340 raft_consensus.cc:740] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.554750 9340 consensus_queue.cc:260] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.568241 9340 ts_tablet_manager.cc:1434] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.028s user 0.002s sys 0.003s
I20260926 01:56:13.569278 9340 tablet_bootstrap.cc:492] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:13.584034 9341 tablet_bootstrap.cc:492] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:13.584635 9341 ts_tablet_manager.cc:1403] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.395s user 0.016s sys 0.000s
I20260926 01:56:13.588057 9341 raft_consensus.cc:359] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.588845 9341 raft_consensus.cc:385] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.589154 9341 raft_consensus.cc:740] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.591866 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:13.592501 8897 raft_consensus.cc:3060] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:13.601648 8897 raft_consensus.cc:2468] T 215a061440854c56b00df9fda9192598 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:13.624346 10478 raft_consensus.cc:493] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:13.624886 10478 raft_consensus.cc:515] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:13.625639 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:13.626995 10478 leader_election.cc:290] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:13.627715 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e140cf11b8bb49f9a94c0158a2afca4d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:13.628257 9068 raft_consensus.cc:3060] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:13.630896 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:13.637879 9339 tablet_bootstrap.cc:654] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:13.640573 9340 tablet_bootstrap.cc:654] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:13.655385 9068 raft_consensus.cc:2468] T e140cf11b8bb49f9a94c0158a2afca4d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:13.676753 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0558d37a3b5f4bcea43a8da7b99ded09" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.589993 9341 consensus_queue.cc:260] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.712232 9341 ts_tablet_manager.cc:1434] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.127s user 0.005s sys 0.000s
W20260926 01:56:13.713794 9006 leader_election.cc:343] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.760722 9003 leader_election.cc:304] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:13.777529 10687 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:13.778497 9339 tablet_bootstrap.cc:492] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:13.773314 9340 tablet_bootstrap.cc:492] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:13.779012 9339 ts_tablet_manager.cc:1403] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.396s user 0.011s sys 0.000s
I20260926 01:56:13.783869 9341 tablet_bootstrap.cc:492] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:13.778414 10687 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:13.788056 9340 ts_tablet_manager.cc:1403] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.218s user 0.018s sys 0.014s
I20260926 01:56:13.799115 10687 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:13.799754 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:13.807708 10621 raft_consensus.cc:2804] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:13.808421 10621 raft_consensus.cc:697] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:13.809376 10621 consensus_queue.cc:237] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.813663 9339 raft_consensus.cc:359] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.814365 9339 raft_consensus.cc:385] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.814633 9339 raft_consensus.cc:740] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.815248 9339 consensus_queue.cc:260] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.819393 9339 ts_tablet_manager.cc:1434] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20260926 01:56:13.820322 9339 tablet_bootstrap.cc:492] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:13.826220 9339 tablet_bootstrap.cc:654] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:13.837502 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 76867.7us
I20260926 01:56:13.841856 9339 tablet_bootstrap.cc:492] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:13.842300 9339 ts_tablet_manager.cc:1403] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20260926 01:56:13.849498 9339 raft_consensus.cc:359] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.850204 9339 raft_consensus.cc:385] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.850471 9339 raft_consensus.cc:740] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.851105 9339 consensus_queue.cc:260] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.854204 9339 ts_tablet_manager.cc:1434] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260926 01:56:13.855173 9339 tablet_bootstrap.cc:492] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:13.861124 9339 tablet_bootstrap.cc:654] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:13.892521 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0558d37a3b5f4bcea43a8da7b99ded09" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:13.899168 9339 tablet_bootstrap.cc:492] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:13.899734 9339 ts_tablet_manager.cc:1403] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.000s
I20260926 01:56:13.902766 9339 raft_consensus.cc:359] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.903430 9339 raft_consensus.cc:385] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.903697 9339 raft_consensus.cc:740] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:13.909420 9339 consensus_queue.cc:260] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.911597 9339 ts_tablet_manager.cc:1434] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260926 01:56:13.916621 9339 tablet_bootstrap.cc:492] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:13.918887 10685 raft_consensus.cc:493] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:13.919647 10685 raft_consensus.cc:515] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.922693 10685 leader_election.cc:290] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:13.924446 8843 leader_election.cc:343] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.924888 8843 leader_election.cc:304] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:13.925983 10653 raft_consensus.cc:2749] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:13.931540 8843 leader_election.cc:343] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:13.931986 8843 leader_election.cc:304] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:13.933280 10653 raft_consensus.cc:2749] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:13.886272 9340 raft_consensus.cc:359] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:13.936432 8843 leader_election.cc:304] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:13.937616 10653 raft_consensus.cc:2804] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:13.938329 10653 raft_consensus.cc:697] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:13.939479 10653 consensus_queue.cc:237] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:13.938092 9340 raft_consensus.cc:385] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:13.948537 9339 tablet_bootstrap.cc:654] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:13.949034 10340 consensus_queue.cc:1048] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:13.948740 9340 raft_consensus.cc:740] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.007477 10340 raft_consensus.cc:493] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:14.008092 10340 raft_consensus.cc:515] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.009977 10340 leader_election.cc:290] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:14.012967 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:13.998242 9340 consensus_queue.cc:260] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.024899 8901 raft_consensus.cc:2468] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:14.027976 10652 raft_consensus.cc:493] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:14.028455 10652 raft_consensus.cc:515] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.032311 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:14.031633 10652 leader_election.cc:290] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:14.033771 8843 leader_election.cc:343] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:14.051885 9341 tablet_bootstrap.cc:654] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.046200 9340 ts_tablet_manager.cc:1434] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.245s user 0.019s sys 0.006s
I20260926 01:56:14.080551 9339 tablet_bootstrap.cc:492] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:14.081051 9339 ts_tablet_manager.cc:1403] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.169s user 0.024s sys 0.040s
I20260926 01:56:14.086627 9339 raft_consensus.cc:359] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.100780 9339 raft_consensus.cc:385] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.101132 9339 raft_consensus.cc:740] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.101855 9339 consensus_queue.cc:260] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.107510 9339 ts_tablet_manager.cc:1434] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260926 01:56:14.111624 9068 raft_consensus.cc:3060] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:14.112152 9339 tablet_bootstrap.cc:492] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:14.124640 9339 tablet_bootstrap.cc:654] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.150982 9340 tablet_bootstrap.cc:492] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:14.150979 9339 tablet_bootstrap.cc:492] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:14.151448 9339 ts_tablet_manager.cc:1403] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.007s
I20260926 01:56:14.153971 9339 raft_consensus.cc:359] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.154701 9339 raft_consensus.cc:385] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.154959 9339 raft_consensus.cc:740] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.155555 9339 consensus_queue.cc:260] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.160142 9340 tablet_bootstrap.cc:654] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.168591 9339 ts_tablet_manager.cc:1434] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260926 01:56:14.169634 9339 tablet_bootstrap.cc:492] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:14.175756 9339 tablet_bootstrap.cc:654] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.169730 9341 tablet_bootstrap.cc:492] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:14.178953 9341 ts_tablet_manager.cc:1403] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.395s user 0.010s sys 0.002s
I20260926 01:56:14.197729 10700 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:14.198583 10700 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:14.209631 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:14.233098 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:14.260128 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:14.260804 8901 raft_consensus.cc:2468] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:14.260172 10652 consensus_queue.cc:1048] T 030f4d683e404fbd8d2e6863e18f71c0 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260926 01:56:14.263226 9341 raft_consensus.cc:359] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.263939 9341 raft_consensus.cc:385] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.264202 9341 raft_consensus.cc:740] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.264985 9341 consensus_queue.cc:260] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.271951 9339 tablet_bootstrap.cc:492] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:14.272410 9339 ts_tablet_manager.cc:1403] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.004s
I20260926 01:56:14.283762 10700 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:14.285300 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
W20260926 01:56:14.315531 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 55677.8us
I20260926 01:56:14.326929 10496 raft_consensus.cc:493] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:14.327466 10496 raft_consensus.cc:515] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.331024 10496 leader_election.cc:290] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:14.332863 9341 ts_tablet_manager.cc:1434] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.153s user 0.005s sys 0.000s
I20260926 01:56:14.333665 10496 raft_consensus.cc:493] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:14.335223 10496 raft_consensus.cc:515] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.337662 10496 leader_election.cc:290] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:14.340341 10662 consensus_queue.cc:1048] T 6ab082379974486bbea9f87f5aa64e10 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:14.347754 9155 leader_election.cc:343] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: bcb000f9234d4382ad019f63015b92e7
I20260926 01:56:14.349681 9340 tablet_bootstrap.cc:492] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:14.350178 9340 ts_tablet_manager.cc:1403] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.199s user 0.010s sys 0.007s
I20260926 01:56:14.350217 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:14.353487 9339 raft_consensus.cc:359] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.354153 9339 raft_consensus.cc:385] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.354408 9339 raft_consensus.cc:740] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.355012 9339 consensus_queue.cc:260] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:14.370890 9006 leader_election.cc:343] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:14.361405 8843 leader_election.cc:343] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: bcb000f9234d4382ad019f63015b92e7
I20260926 01:56:14.372223 9339 ts_tablet_manager.cc:1434] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.100s user 0.007s sys 0.000s
I20260926 01:56:14.374811 8843 leader_election.cc:304] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.387518 9340 raft_consensus.cc:359] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.388293 9340 raft_consensus.cc:385] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.388578 9340 raft_consensus.cc:740] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.389082 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:14.389678 8904 raft_consensus.cc:2468] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:14.390377 10662 raft_consensus.cc:2749] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:14.388306 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f7419abcc2c482e97df70f03e047450" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:14.407011 9341 tablet_bootstrap.cc:492] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:14.410773 9003 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 47927.6us
I20260926 01:56:14.412961 9339 tablet_bootstrap.cc:492] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:14.422183 8843 leader_election.cc:343] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:14.434271 9003 leader_election.cc:304] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:14.436440 8843 leader_election.cc:304] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.427141 9340 consensus_queue.cc:260] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.447724 10635 raft_consensus.cc:493] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:14.448494 10702 raft_consensus.cc:2749] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:14.448254 10635 raft_consensus.cc:515] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.450264 10635 leader_election.cc:290] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:14.451625 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:14.454226 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 49019.2us
I20260926 01:56:14.467337 10630 raft_consensus.cc:2804] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:14.467836 10630 raft_consensus.cc:493] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:14.468158 10630 raft_consensus.cc:3060] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:14.470974 9340 ts_tablet_manager.cc:1434] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.120s user 0.005s sys 0.005s
I20260926 01:56:14.479467 10630 raft_consensus.cc:515] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.482019 9217 raft_consensus.cc:3060] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:14.508628 10630 leader_election.cc:290] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:14.486879 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 52700.1us
I20260926 01:56:14.513175 9003 leader_election.cc:304] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:14.514190 10621 raft_consensus.cc:2804] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:14.514582 10621 raft_consensus.cc:493] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:14.514882 10621 raft_consensus.cc:3060] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:14.524642 10621 raft_consensus.cc:515] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.544770 9339 tablet_bootstrap.cc:654] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.548856 9340 tablet_bootstrap.cc:492] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:14.555181 9340 tablet_bootstrap.cc:654] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.556370 9152 leader_election.cc:304] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.563694 10635 raft_consensus.cc:2804] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:56:14.564152 10635 raft_consensus.cc:493] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:14.564504 10635 raft_consensus.cc:3060] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:14.577001 9341 tablet_bootstrap.cc:654] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.590992 10635 raft_consensus.cc:515] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:14.592551 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 79463.3us
W20260926 01:56:14.602392 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 46141.8us
I20260926 01:56:14.603282 9152 leader_election.cc:304] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.604202 10496 raft_consensus.cc:2749] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:14.604918 10684 consensus_queue.cc:1048] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:14.611732 10635 leader_election.cc:290] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:14.613967 9155 leader_election.cc:343] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: bcb000f9234d4382ad019f63015b92e7
I20260926 01:56:14.624337 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0558d37a3b5f4bcea43a8da7b99ded09" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:14.624984 8901 raft_consensus.cc:3060] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:14.627322 9340 tablet_bootstrap.cc:492] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:14.627800 9340 ts_tablet_manager.cc:1403] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.000s
I20260926 01:56:14.631466 9340 raft_consensus.cc:359] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.632339 9340 raft_consensus.cc:385] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.632596 9340 raft_consensus.cc:740] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.636632 8901 raft_consensus.cc:2468] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:14.637233 10621 leader_election.cc:290] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:14.643879 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c26f557ac274973824e6f9a4532de74" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:14.661561 9341 tablet_bootstrap.cc:492] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:14.662029 9341 ts_tablet_manager.cc:1403] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.255s user 0.004s sys 0.006s
I20260926 01:56:14.663856 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:14.665335 9210 raft_consensus.cc:3060] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:14.665526 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcb000f9234d4382ad019f63015b92e7" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:14.666157 8901 raft_consensus.cc:3060] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:14.666808 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0558d37a3b5f4bcea43a8da7b99ded09" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:14.682703 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:14.683372 8897 raft_consensus.cc:3060] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:14.694180 8897 raft_consensus.cc:2468] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:14.697690 9339 tablet_bootstrap.cc:492] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:14.698287 9339 ts_tablet_manager.cc:1403] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.299s user 0.008s sys 0.005s
I20260926 01:56:14.707458 9152 leader_election.cc:304] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.718816 9340 consensus_queue.cc:260] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.720968 9340 ts_tablet_manager.cc:1434] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.093s user 0.006s sys 0.001s
I20260926 01:56:14.709400 10684 raft_consensus.cc:493] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:14.722234 10684 raft_consensus.cc:515] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.726610 9339 raft_consensus.cc:359] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.727706 9339 raft_consensus.cc:385] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.727969 9339 raft_consensus.cc:740] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.729193 9341 raft_consensus.cc:359] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.729874 9341 raft_consensus.cc:385] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.730118 9341 raft_consensus.cc:740] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.730772 9341 consensus_queue.cc:260] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.731315 10708 raft_consensus.cc:2749] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:14.741403 10684 leader_election.cc:290] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:14.751256 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 107141us
I20260926 01:56:14.754946 9341 ts_tablet_manager.cc:1434] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.093s user 0.008s sys 0.000s
W20260926 01:56:14.755456 8524 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet cbe24f2921db43ff987ca93cbdad2f42 (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet cbe24f2921db43ff987ca93cbdad2f42 consensus state has no leader
I20260926 01:56:14.756035 9341 tablet_bootstrap.cc:492] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:14.759150 9339 consensus_queue.cc:260] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:14.763628 8843 consensus_peers.cc:597] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:14.767390 8901 raft_consensus.cc:2468] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 2.
I20260926 01:56:14.777215 9152 leader_election.cc:304] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters: d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.781574 10635 raft_consensus.cc:2804] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:56:14.782186 10635 raft_consensus.cc:697] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 2 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:14.784344 10635 consensus_queue.cc:237] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.790081 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d198e51b88948548cfd95dd893a26ef" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:14.805828 9340 tablet_bootstrap.cc:492] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:14.808158 10708 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:14.808660 10708 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.810707 10708 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:14.818244 9152 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
W20260926 01:56:14.821487 9155 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:14.822191 9155 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.823060 10708 raft_consensus.cc:2749] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:14.825057 10710 raft_consensus.cc:493] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:14.825587 10710 raft_consensus.cc:515] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.827622 10710 leader_election.cc:290] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:14.830681 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 76786us
W20260926 01:56:14.833649 9006 leader_election.cc:343] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:14.838572 9339 ts_tablet_manager.cc:1434] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.117s user 0.007s sys 0.000s
I20260926 01:56:14.839743 9339 tablet_bootstrap.cc:492] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:14.843222 9003 leader_election.cc:304] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:14.844174 9340 tablet_bootstrap.cc:654] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.844242 10685 consensus_queue.cc:1048] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:14.844947 10710 raft_consensus.cc:2804] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:14.845513 10710 raft_consensus.cc:697] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:14.846403 10710 consensus_queue.cc:237] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.871166 9341 tablet_bootstrap.cc:654] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:14.882825 8599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.594s (SENT) [suppressed 1 similar messages]
W20260926 01:56:14.885682 9006 leader_election.cc:343] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:14.910683 9340 tablet_bootstrap.cc:492] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:14.911293 9340 ts_tablet_manager.cc:1403] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.106s user 0.011s sys 0.002s
I20260926 01:56:14.914319 9340 raft_consensus.cc:359] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.930804 9003 leader_election.cc:304] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:14.931685 10684 raft_consensus.cc:2804] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:56:14.931818 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 82419.8us
I20260926 01:56:14.932171 10684 raft_consensus.cc:697] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
W20260926 01:56:14.930812 9006 leader_election.cc:343] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:14.934530 10684 consensus_queue.cc:237] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.938617 10630 consensus_queue.cc:1048] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:14.940440 9339 tablet_bootstrap.cc:654] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:14.948017 9340 raft_consensus.cc:385] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:14.950357 9340 raft_consensus.cc:740] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:14.953230 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b612e8955d5f42dd8658320031fe36ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:14.947366 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d198e51b88948548cfd95dd893a26ef" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:14.966612 10720 raft_consensus.cc:493] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:14.967532 10720 raft_consensus.cc:515] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:14.970883 10720 leader_election.cc:290] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:14.972474 9152 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:14.974772 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b612e8955d5f42dd8658320031fe36ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:14.978848 9155 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:14.979205 9155 leader_election.cc:304] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:14.982349 10635 raft_consensus.cc:2749] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:14.988008 9006 consensus_peers.cc:597] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:14.977099 9340 consensus_queue.cc:260] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:14.989296 8907 raft_consensus.cc:2468] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:56:14.988803 9006 leader_election.cc:343] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:14.992002 10621 raft_consensus.cc:493] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:14.992448 10621 raft_consensus.cc:515] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:14.994398 10621 leader_election.cc:290] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:15.000452 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:15.001746 9341 tablet_bootstrap.cc:492] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:15.002360 9341 ts_tablet_manager.cc:1403] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.247s user 0.010s sys 0.002s
W20260926 01:56:15.004873 9006 leader_election.cc:343] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.008137 9340 ts_tablet_manager.cc:1434] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.096s user 0.007s sys 0.000s
I20260926 01:56:15.012342 9340 tablet_bootstrap.cc:492] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:15.021822 9339 tablet_bootstrap.cc:492] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:15.022598 9339 ts_tablet_manager.cc:1403] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.183s user 0.005s sys 0.009s
I20260926 01:56:15.027803 9339 raft_consensus.cc:359] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.028486 9339 raft_consensus.cc:385] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.028741 9339 raft_consensus.cc:740] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.029389 9339 consensus_queue.cc:260] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.031255 9339 ts_tablet_manager.cc:1434] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260926 01:56:15.032390 9339 tablet_bootstrap.cc:492] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:15.036409 9341 raft_consensus.cc:359] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.060786 9339 tablet_bootstrap.cc:654] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.063876 10702 consensus_queue.cc:1048] T e140cf11b8bb49f9a94c0158a2afca4d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:15.064045 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:15.066471 8897 raft_consensus.cc:2468] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:56:15.069545 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.569s (SENT)
I20260926 01:56:15.070024 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:15.068703 9340 tablet_bootstrap.cc:654] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.037196 9341 raft_consensus.cc:385] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.092553 9341 raft_consensus.cc:740] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.095775 9341 consensus_queue.cc:260] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.097880 10734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:15.106499 9339 tablet_bootstrap.cc:492] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:15.107064 9339 ts_tablet_manager.cc:1403] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.014s
I20260926 01:56:15.111497 9339 raft_consensus.cc:359] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:15.112156 9339 raft_consensus.cc:385] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.112404 9339 raft_consensus.cc:740] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.113025 9339 consensus_queue.cc:260] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:15.115254 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 45364.6us
W20260926 01:56:15.146692 9003 leader_election.cc:343] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.180061 9340 tablet_bootstrap.cc:492] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:15.180542 9340 ts_tablet_manager.cc:1403] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.168s user 0.008s sys 0.004s
I20260926 01:56:15.182958 9341 ts_tablet_manager.cc:1434] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.180s user 0.006s sys 0.001s
I20260926 01:56:15.183245 9340 raft_consensus.cc:359] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.183903 9340 raft_consensus.cc:385] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.184185 9340 raft_consensus.cc:740] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.186942 9339 ts_tablet_manager.cc:1434] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.080s user 0.003s sys 0.004s
I20260926 01:56:15.147800 9003 leader_election.cc:304] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.184823 9340 consensus_queue.cc:260] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.193334 10685 raft_consensus.cc:2749] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:15.194501 9340 ts_tablet_manager.cc:1434] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260926 01:56:15.195518 9340 tablet_bootstrap.cc:492] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:15.198529 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 51883.6us
I20260926 01:56:15.206161 9003 leader_election.cc:304] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.207019 10685 raft_consensus.cc:2804] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:15.207427 10685 raft_consensus.cc:493] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:15.207823 10685 raft_consensus.cc:3060] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.218158 10685 raft_consensus.cc:515] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:15.236923 9341 tablet_bootstrap.cc:492] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:15.220023 9339 tablet_bootstrap.cc:492] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:15.254906 9339 tablet_bootstrap.cc:654] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.274884 9340 tablet_bootstrap.cc:654] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.278113 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b612e8955d5f42dd8658320031fe36ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:15.292440 10738 raft_consensus.cc:493] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:15.233784 9003 leader_election.cc:304] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.293829 10684 raft_consensus.cc:2804] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260926 01:56:15.279579 9006 leader_election.cc:343] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.294477 10684 raft_consensus.cc:493] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:15.294857 10684 raft_consensus.cc:3060] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.294711 10738 raft_consensus.cc:515] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.298372 10738 leader_election.cc:290] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:15.299075 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 65386.8us
I20260926 01:56:15.299937 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:15.302152 8843 leader_election.cc:343] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.318594 10731 raft_consensus.cc:493] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:15.319165 10731 raft_consensus.cc:515] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.321277 10731 leader_election.cc:290] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:15.275978 10685 leader_election.cc:290] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:15.331332 10684 raft_consensus.cc:515] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.326640 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b612e8955d5f42dd8658320031fe36ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:15.333415 8907 raft_consensus.cc:3060] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.345768 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d198e51b88948548cfd95dd893a26ef" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:15.346323 10731 raft_consensus.cc:493] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:15.348202 10731 raft_consensus.cc:515] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.349037 9341 tablet_bootstrap.cc:654] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.350445 10731 leader_election.cc:290] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:15.347707 8843 leader_election.cc:343] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.356385 9340 tablet_bootstrap.cc:492] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:15.356956 9340 ts_tablet_manager.cc:1403] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.162s user 0.013s sys 0.001s
I20260926 01:56:15.360942 8907 raft_consensus.cc:2468] T b612e8955d5f42dd8658320031fe36ca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:15.386142 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:15.387019 9069 raft_consensus.cc:2468] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:15.371788 9340 raft_consensus.cc:359] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:15.396353 9340 raft_consensus.cc:385] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.396690 9340 raft_consensus.cc:740] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.398994 9340 consensus_queue.cc:260] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:15.402665 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec749c375d784786ba93020996708bb1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:15.404343 8843 leader_election.cc:343] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.405630 9340 ts_tablet_manager.cc:1434] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.048s user 0.005s sys 0.001s
I20260926 01:56:15.406692 9340 tablet_bootstrap.cc:492] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:15.434957 9340 tablet_bootstrap.cc:654] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.435153 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d198e51b88948548cfd95dd893a26ef" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:15.436327 9069 raft_consensus.cc:2468] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:15.452212 10684 leader_election.cc:290] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:15.452962 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:15.454402 9006 leader_election.cc:343] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.457757 9339 tablet_bootstrap.cc:492] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:15.458264 9339 ts_tablet_manager.cc:1403] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.238s user 0.009s sys 0.003s
I20260926 01:56:15.462702 9341 tablet_bootstrap.cc:492] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:15.463328 9341 ts_tablet_manager.cc:1403] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.227s user 0.008s sys 0.003s
I20260926 01:56:15.463016 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec749c375d784786ba93020996708bb1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:15.463883 9069 raft_consensus.cc:2468] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:15.465008 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:15.465788 8900 raft_consensus.cc:3060] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.465889 9341 raft_consensus.cc:359] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.485525 8843 leader_election.cc:304] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.486398 10731 raft_consensus.cc:2804] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:15.486750 10731 raft_consensus.cc:493] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:15.487066 10731 raft_consensus.cc:3060] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.495682 9341 raft_consensus.cc:385] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.496011 9341 raft_consensus.cc:740] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.495702 10731 raft_consensus.cc:515] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.496801 9341 consensus_queue.cc:260] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.498082 10731 leader_election.cc:290] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:15.499123 9341 ts_tablet_manager.cc:1434] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.036s user 0.003s sys 0.003s
W20260926 01:56:15.503388 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.594s (SENT)
I20260926 01:56:15.503858 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:15.504212 10635 raft_consensus.cc:493] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:15.504698 10635 raft_consensus.cc:515] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:15.505740 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:15.506750 10635 leader_election.cc:290] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:15.511052 8843 leader_election.cc:343] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.517283 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0558d37a3b5f4bcea43a8da7b99ded09" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:15.518190 8907 raft_consensus.cc:2393] T 0558d37a3b5f4bcea43a8da7b99ded09 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:15.518819 10684 raft_consensus.cc:493] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:15.519459 10684 raft_consensus.cc:515] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.519861 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0558d37a3b5f4bcea43a8da7b99ded09" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:15.521891 10684 leader_election.cc:290] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:15.521973 9155 leader_election.cc:304] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:15.522521 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:15.525733 10635 raft_consensus.cc:3060] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:15.530901 9006 leader_election.cc:343] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:15.532436 9340 tablet_bootstrap.cc:492] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:15.532958 9340 ts_tablet_manager.cc:1403] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.002s
I20260926 01:56:15.535496 10635 raft_consensus.cc:2749] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:15.536016 9340 raft_consensus.cc:359] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.537307 9340 raft_consensus.cc:385] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.537619 9340 raft_consensus.cc:740] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.538697 9340 consensus_queue.cc:260] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.540195 9340 ts_tablet_manager.cc:1434] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:56:15.561465 9339 raft_consensus.cc:359] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.564670 10747 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:15.569664 9339 raft_consensus.cc:385] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 01:56:15.582406 9155 consensus_peers.cc:597] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcb000f9234d4382ad019f63015b92e7. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:15.584144 9339 raft_consensus.cc:740] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.585767 9339 consensus_queue.cc:260] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.624104 10684 raft_consensus.cc:493] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:15.624686 10684 raft_consensus.cc:515] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.627081 10684 leader_election.cc:290] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:15.563087 8843 leader_election.cc:304] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
W20260926 01:56:15.631122 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 68162.7us
I20260926 01:56:15.631711 10738 raft_consensus.cc:2804] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:15.633464 10738 raft_consensus.cc:493] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:15.633841 10738 raft_consensus.cc:3060] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.648809 10738 raft_consensus.cc:515] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.651232 9003 leader_election.cc:304] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.652596 10685 raft_consensus.cc:2804] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:15.653160 10685 raft_consensus.cc:697] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:15.654400 10685 consensus_queue.cc:237] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:15.662352 9214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8711788a85f94c599224123f6b8cc814" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:15.663061 9214 raft_consensus.cc:2468] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:15.669621 9006 leader_election.cc:304] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:15.670491 10710 raft_consensus.cc:2804] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:15.670843 10710 raft_consensus.cc:493] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:15.671154 10710 raft_consensus.cc:3060] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.679019 10710 raft_consensus.cc:515] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.687989 9341 tablet_bootstrap.cc:492] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:15.690814 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8711788a85f94c599224123f6b8cc814" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:15.691442 9220 raft_consensus.cc:3060] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.697851 9339 ts_tablet_manager.cc:1434] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.239s user 0.004s sys 0.002s
I20260926 01:56:15.702030 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d198e51b88948548cfd95dd893a26ef" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:15.702680 9218 raft_consensus.cc:3060] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.707870 9341 tablet_bootstrap.cc:654] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.710851 10710 leader_election.cc:290] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:15.713670 9218 raft_consensus.cc:2468] T 8d198e51b88948548cfd95dd893a26ef P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:15.717082 9220 raft_consensus.cc:2468] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:15.718433 9006 leader_election.cc:304] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:15.719614 8843 leader_election.cc:304] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:15.720391 10702 raft_consensus.cc:2804] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:15.720846 10702 raft_consensus.cc:697] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:15.722046 10684 raft_consensus.cc:2804] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:15.721614 10702 consensus_queue.cc:237] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.722645 10684 raft_consensus.cc:697] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:15.723436 10684 consensus_queue.cc:237] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.740872 9339 tablet_bootstrap.cc:492] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:15.741817 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 90670.7us
I20260926 01:56:15.746706 8900 raft_consensus.cc:2468] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:15.764720 9340 tablet_bootstrap.cc:492] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:15.769335 10738 leader_election.cc:290] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:15.770051 10710 consensus_queue.cc:1048] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:15.777057 9003 leader_election.cc:304] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.798560 10621 raft_consensus.cc:2804] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:15.799161 10621 raft_consensus.cc:697] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:15.800036 10621 consensus_queue.cc:237] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.809705 9339 tablet_bootstrap.cc:654] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:15.814136 9006 consensus_peers.cc:597] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:15.815255 8843 leader_election.cc:304] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.821221 9341 tablet_bootstrap.cc:492] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:15.821663 9341 ts_tablet_manager.cc:1403] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.134s user 0.010s sys 0.001s
I20260926 01:56:15.844772 10757 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:15.847864 10757 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:15.850266 10757 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:15.853036 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:15.858964 9339 tablet_bootstrap.cc:492] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:15.859532 9339 ts_tablet_manager.cc:1403] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.122s user 0.016s sys 0.000s
I20260926 01:56:15.861712 9341 raft_consensus.cc:359] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.862649 9341 raft_consensus.cc:385] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.862972 9341 raft_consensus.cc:740] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.862066 9339 raft_consensus.cc:359] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.875028 9339 raft_consensus.cc:385] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.875397 9339 raft_consensus.cc:740] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.876224 9339 consensus_queue.cc:260] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.884686 9339 ts_tablet_manager.cc:1434] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260926 01:56:15.885717 9339 tablet_bootstrap.cc:492] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:15.866602 10738 raft_consensus.cc:2804] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:15.911744 9339 tablet_bootstrap.cc:654] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.913727 9341 consensus_queue.cc:260] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.916721 9341 ts_tablet_manager.cc:1434] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.095s user 0.007s sys 0.000s
I20260926 01:56:15.937948 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:15.938737 8897 raft_consensus.cc:2468] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:15.895515 10738 raft_consensus.cc:493] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:15.941617 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:15.942251 9069 raft_consensus.cc:3060] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:15.948876 9339 tablet_bootstrap.cc:492] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:15.949373 9339 ts_tablet_manager.cc:1403] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.000s
I20260926 01:56:15.960601 9339 raft_consensus.cc:359] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.961318 9339 raft_consensus.cc:385] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:15.961607 9339 raft_consensus.cc:740] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:15.967886 9340 tablet_bootstrap.cc:654] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:15.962240 9339 consensus_queue.cc:260] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:15.975247 9339 ts_tablet_manager.cc:1434] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260926 01:56:15.980191 9069 raft_consensus.cc:2468] T 2c0cd3e30f084966b27921e658cacf58 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:16.002246 10635 raft_consensus.cc:493] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:16.002784 10635 raft_consensus.cc:515] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.003126 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d198e51b88948548cfd95dd893a26ef" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:16.003921 9068 raft_consensus.cc:3060] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:16.005481 10635 leader_election.cc:290] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:16.006652 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:16.016295 9068 raft_consensus.cc:2468] T 8d198e51b88948548cfd95dd893a26ef P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:16.033989 8843 leader_election.cc:304] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:15.951438 10738 raft_consensus.cc:3060] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:16.054961 9341 tablet_bootstrap.cc:492] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:16.038246 9339 tablet_bootstrap.cc:492] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:16.061659 9341 tablet_bootstrap.cc:654] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.062870 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8711788a85f94c599224123f6b8cc814" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:16.053248 10731 raft_consensus.cc:2804] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:16.066541 10731 raft_consensus.cc:697] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:16.067409 10731 consensus_queue.cc:237] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.073586 8902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8711788a85f94c599224123f6b8cc814" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:16.080185 8900 raft_consensus.cc:2468] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:16.089519 8897 raft_consensus.cc:1275] T bcb000f9234d4382ad019f63015b92e7 P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Refusing update from remote peer bc00feec3f564608851f0112e97ab532: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260926 01:56:16.097795 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030c41a9718d47ddaa33382b20a60dab" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:16.098766 8905 raft_consensus.cc:2393] T 030c41a9718d47ddaa33382b20a60dab P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:16.107882 10738 raft_consensus.cc:515] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:16.112303 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 64584.7us
I20260926 01:56:16.118249 10635 consensus_queue.cc:1048] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260926 01:56:16.121896 9339 tablet_bootstrap.cc:654] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.122612 10684 raft_consensus.cc:493] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:16.123065 10684 raft_consensus.cc:515] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:16.126672 10684 leader_election.cc:290] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:16.131899 9341 tablet_bootstrap.cc:492] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:16.132369 9341 ts_tablet_manager.cc:1403] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.001s
I20260926 01:56:16.133524 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:16.136394 9006 leader_election.cc:343] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:16.134989 9341 raft_consensus.cc:359] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.144064 9341 raft_consensus.cc:385] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.144357 9341 raft_consensus.cc:740] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.146386 9341 consensus_queue.cc:260] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.167218 9339 tablet_bootstrap.cc:492] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:16.167681 9339 ts_tablet_manager.cc:1403] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.130s user 0.014s sys 0.000s
I20260926 01:56:16.171598 9341 ts_tablet_manager.cc:1434] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.039s user 0.006s sys 0.003s
I20260926 01:56:16.174818 9339 raft_consensus.cc:359] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:16.175565 9339 raft_consensus.cc:385] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.175837 9339 raft_consensus.cc:740] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.176509 9339 consensus_queue.cc:260] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:16.183621 9339 ts_tablet_manager.cc:1434] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260926 01:56:16.184767 9339 tablet_bootstrap.cc:492] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:16.163256 9340 tablet_bootstrap.cc:492] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:16.200752 9340 ts_tablet_manager.cc:1403] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.436s user 0.008s sys 0.007s
I20260926 01:56:16.201552 10684 raft_consensus.cc:493] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:16.202113 10684 raft_consensus.cc:515] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.204360 10684 leader_election.cc:290] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:16.207033 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6e8059b94564c42894c6c8a3d695950" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:16.209726 9006 leader_election.cc:343] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:16.211992 9339 tablet_bootstrap.cc:654] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.214074 9340 raft_consensus.cc:359] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.214823 9340 raft_consensus.cc:385] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.215128 9340 raft_consensus.cc:740] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.215821 9340 consensus_queue.cc:260] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.220500 9152 leader_election.cc:304] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:16.221262 10748 raft_consensus.cc:3060] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:16.229069 10748 raft_consensus.cc:2749] T 030c41a9718d47ddaa33382b20a60dab P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:16.244290 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec749c375d784786ba93020996708bb1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:16.245918 8843 leader_election.cc:343] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:16.251245 10731 raft_consensus.cc:493] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:16.251760 10731 raft_consensus.cc:515] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:16.253469 9339 tablet_bootstrap.cc:492] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:16.254626 10731 leader_election.cc:290] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:16.256402 9339 ts_tablet_manager.cc:1403] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.072s user 0.014s sys 0.009s
I20260926 01:56:16.258975 9339 raft_consensus.cc:359] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.259620 9339 raft_consensus.cc:385] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.259454 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:16.259884 9339 raft_consensus.cc:740] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.260494 9339 consensus_queue.cc:260] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:16.262168 8843 leader_election.cc:343] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:16.262504 9339 ts_tablet_manager.cc:1434] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 01:56:16.263428 9339 tablet_bootstrap.cc:492] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:16.269505 9339 tablet_bootstrap.cc:654] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.279177 10738 leader_election.cc:290] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:16.282073 9003 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 216786us
W20260926 01:56:16.348387 9003 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 54732.3us
I20260926 01:56:16.217357 9340 ts_tablet_manager.cc:1434] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260926 01:56:16.359170 9340 tablet_bootstrap.cc:492] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:16.370850 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:16.371481 8905 raft_consensus.cc:2468] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:16.375053 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6e8059b94564c42894c6c8a3d695950" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:16.375710 8897 raft_consensus.cc:2468] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:56:16.413374 9155 consensus_peers.cc:597] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16f49a5a10ea4f7a821d89f18e3af560. This is attempt 6: this message will repeat every 5th retry.
I20260926 01:56:16.422920 9339 tablet_bootstrap.cc:492] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:16.423466 9339 ts_tablet_manager.cc:1403] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.160s user 0.030s sys 0.042s
I20260926 01:56:16.426710 9341 tablet_bootstrap.cc:492] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:16.428200 9003 leader_election.cc:304] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:16.428965 10755 raft_consensus.cc:2804] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:16.429291 10755 raft_consensus.cc:493] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:16.429595 10755 raft_consensus.cc:3060] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:16.436461 9341 tablet_bootstrap.cc:654] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.436801 10755 raft_consensus.cc:515] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.446859 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:16.448148 9006 leader_election.cc:343] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:16.446241 10755 leader_election.cc:290] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:16.426105 9339 raft_consensus.cc:359] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.459179 9339 raft_consensus.cc:385] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.459533 9339 raft_consensus.cc:740] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.460210 9339 consensus_queue.cc:260] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.461145 10635 raft_consensus.cc:493] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:16.461593 10635 raft_consensus.cc:515] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.463755 10635 leader_election.cc:290] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:16.464552 9339 ts_tablet_manager.cc:1434] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20260926 01:56:16.464910 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:16.466267 9339 tablet_bootstrap.cc:492] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:16.478832 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "215a061440854c56b00df9fda9192598" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:16.480396 9152 leader_election.cc:304] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:16.481562 10635 raft_consensus.cc:2749] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:16.490989 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 62874.8us
W20260926 01:56:16.508100 9006 consensus_peers.cc:597] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:16.518294 10778 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:16.490266 9340 tablet_bootstrap.cc:654] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.526814 10778 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:16.547406 9341 tablet_bootstrap.cc:492] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:16.547904 9341 ts_tablet_manager.cc:1403] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.003s
I20260926 01:56:16.552935 9341 raft_consensus.cc:359] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.558588 9341 raft_consensus.cc:385] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.559741 9341 raft_consensus.cc:740] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.561419 9341 consensus_queue.cc:260] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.567154 9341 ts_tablet_manager.cc:1434] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.019s user 0.008s sys 0.003s
I20260926 01:56:16.573292 9341 tablet_bootstrap.cc:492] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:16.580803 9341 tablet_bootstrap.cc:654] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.583182 10778 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:16.584069 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:16.582840 9340 tablet_bootstrap.cc:492] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:16.589188 9340 ts_tablet_manager.cc:1403] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.230s user 0.010s sys 0.003s
W20260926 01:56:16.619553 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 123596us
I20260926 01:56:16.630667 10731 consensus_queue.cc:1048] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:16.663430 9340 raft_consensus.cc:359] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.664173 9340 raft_consensus.cc:385] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.664479 9340 raft_consensus.cc:740] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.665139 9340 consensus_queue.cc:260] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.686798 9339 tablet_bootstrap.cc:700] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2: Time spent opening tablet: real 0.148s user 0.002s sys 0.000s
I20260926 01:56:16.687940 9339 tablet_bootstrap.cc:654] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.694432 9341 tablet_bootstrap.cc:492] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:16.694957 9341 ts_tablet_manager.cc:1403] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.122s user 0.010s sys 0.001s
I20260926 01:56:16.690779 9340 ts_tablet_manager.cc:1434] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.080s user 0.007s sys 0.000s
I20260926 01:56:16.699224 9341 raft_consensus.cc:359] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.699930 9341 raft_consensus.cc:385] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.700206 9341 raft_consensus.cc:740] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.702099 9341 consensus_queue.cc:260] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.706245 10731 raft_consensus.cc:493] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:16.706845 10731 raft_consensus.cc:515] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.709812 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:16.711167 8843 leader_election.cc:343] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:16.711839 10731 leader_election.cc:290] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:16.719744 9341 ts_tablet_manager.cc:1434] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
W20260926 01:56:16.739817 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 49431.6us
I20260926 01:56:16.763473 10621 consensus_queue.cc:1048] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:16.768160 10684 consensus_queue.cc:1048] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:16.796043 9340 tablet_bootstrap.cc:492] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:16.799849 9003 leader_election.cc:304] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:16.800926 10621 raft_consensus.cc:2804] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:16.801481 10621 raft_consensus.cc:493] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:16.801911 10621 raft_consensus.cc:3060] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:16.802300 9340 tablet_bootstrap.cc:654] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.812047 9341 tablet_bootstrap.cc:492] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:16.834653 9340 tablet_bootstrap.cc:492] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:16.835146 9340 ts_tablet_manager.cc:1403] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.000s
I20260926 01:56:16.837911 9340 raft_consensus.cc:359] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:16.839679 10720 raft_consensus.cc:493] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:16.840042 9340 raft_consensus.cc:385] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:16.840227 10720 raft_consensus.cc:515] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:16.840651 9340 raft_consensus.cc:740] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:16.841282 9340 consensus_queue.cc:260] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:16.842367 10720 leader_election.cc:290] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:16.845276 9152 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
W20260926 01:56:16.846935 9155 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
I20260926 01:56:16.847354 9155 leader_election.cc:304] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:16.848635 10720 raft_consensus.cc:2749] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:16.850637 9340 ts_tablet_manager.cc:1434] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
W20260926 01:56:16.851444 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 51710.2us
I20260926 01:56:16.851783 9340 tablet_bootstrap.cc:492] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:16.861408 9339 tablet_bootstrap.cc:492] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:16.862006 9339 ts_tablet_manager.cc:1403] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.396s user 0.018s sys 0.008s
W20260926 01:56:16.871392 9006 consensus_peers.cc:597] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:16.882893 9341 tablet_bootstrap.cc:654] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.892118 9340 tablet_bootstrap.cc:654] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:16.854514 9003 leader_election.cc:304] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
W20260926 01:56:16.940486 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 63844.6us
I20260926 01:56:16.945907 10684 raft_consensus.cc:2804] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:16.947465 10684 raft_consensus.cc:493] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:16.949702 10685 raft_consensus.cc:493] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:16.964495 10684 raft_consensus.cc:3060] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:16.964565 10685 raft_consensus.cc:515] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.968797 10685 leader_election.cc:290] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:16.971055 10621 raft_consensus.cc:515] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:16.973075 10621 leader_election.cc:290] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:16.989929 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 135522us
I20260926 01:56:17.002796 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:17.003854 9068 raft_consensus.cc:2393] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:17.004520 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db85241141f4a54a2b7dce15412a6f9" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:17.006784 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:17.009028 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec749c375d784786ba93020996708bb1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:17.009312 9220 raft_consensus.cc:3060] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.009953 9068 raft_consensus.cc:3060] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.013233 10621 raft_consensus.cc:493] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.013747 10621 raft_consensus.cc:515] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:17.015836 10621 leader_election.cc:290] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:17.021407 9339 raft_consensus.cc:359] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.022207 9339 raft_consensus.cc:385] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.022670 9339 raft_consensus.cc:740] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:17.023444 9339 consensus_queue.cc:260] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.025209 9339 ts_tablet_manager.cc:1434] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.163s user 0.006s sys 0.000s
W20260926 01:56:17.034053 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.589s (SENT)
I20260926 01:56:17.034515 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:17.040056 9341 tablet_bootstrap.cc:492] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:17.040598 9341 ts_tablet_manager.cc:1403] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.319s user 0.010s sys 0.001s
I20260926 01:56:17.043485 9341 raft_consensus.cc:359] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.044167 9341 raft_consensus.cc:385] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.044436 9341 raft_consensus.cc:740] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:17.044633 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5c66728bb54ec6beb4f0a2c25d9105" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:17.045125 9341 consensus_queue.cc:260] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:16.942948 10731 raft_consensus.cc:493] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.048619 10731 raft_consensus.cc:515] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.050905 10731 leader_election.cc:290] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:17.052075 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cca4587d64bc4964b298ba1a74fe9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:17.053534 8843 leader_election.cc:343] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.088863 10794 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260926 01:56:17.089653 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 55648.3us
W20260926 01:56:17.093470 8527 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 8a54eb3b8cf64a478df38df9d4aef02f (table default.Table_292982f73b37414ebefe3d3493c525c5 [id=5d7d8e6e469f45829ccc4dbe2085bf09], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8a54eb3b8cf64a478df38df9d4aef02f consensus state has no leader
I20260926 01:56:17.094127 9341 ts_tablet_manager.cc:1434] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20260926 01:56:17.032090 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:17.097559 9066 raft_consensus.cc:2468] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
W20260926 01:56:17.044212 9006 leader_election.cc:343] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:17.100294 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 56137.5us
W20260926 01:56:17.101002 9006 leader_election.cc:343] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:17.102005 9006 leader_election.cc:343] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.104573 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:17.105135 8905 raft_consensus.cc:3060] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:17.106980 8843 consensus_peers.cc:597] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:17.119326 8905 raft_consensus.cc:2468] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:17.085453 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cca4587d64bc4964b298ba1a74fe9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:17.120221 9069 raft_consensus.cc:2468] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:17.124593 9339 tablet_bootstrap.cc:492] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:17.125793 9341 tablet_bootstrap.cc:492] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:17.095094 9068 raft_consensus.cc:2468] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:17.078222 10684 raft_consensus.cc:515] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.133689 10684 leader_election.cc:290] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:17.140839 9219 raft_consensus.cc:3060] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.162680 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6e8059b94564c42894c6c8a3d695950" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:17.124112 9340 tablet_bootstrap.cc:492] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:17.165428 9340 ts_tablet_manager.cc:1403] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.314s user 0.012s sys 0.000s
I20260926 01:56:17.180270 9340 raft_consensus.cc:359] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.181316 9340 raft_consensus.cc:385] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.181597 9340 raft_consensus.cc:740] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
W20260926 01:56:17.187086 9006 leader_election.cc:343] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.190253 9339 tablet_bootstrap.cc:654] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:17.190783 9340 consensus_queue.cc:260] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.197171 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db85241141f4a54a2b7dce15412a6f9" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:17.205636 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:17.206341 8904 raft_consensus.cc:3060] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.210209 9341 tablet_bootstrap.cc:654] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:17.215430 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5c66728bb54ec6beb4f0a2c25d9105" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:17.216185 8900 raft_consensus.cc:2468] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:17.232849 8843 leader_election.cc:304] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:17.236565 9340 ts_tablet_manager.cc:1434] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.071s user 0.003s sys 0.003s
I20260926 01:56:17.237592 9340 tablet_bootstrap.cc:492] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:17.273980 10738 consensus_queue.cc:1048] T 8d198e51b88948548cfd95dd893a26ef P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:17.281780 8904 raft_consensus.cc:2468] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:17.286597 10762 raft_consensus.cc:2749] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:17.299131 10684 consensus_queue.cc:1048] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:17.325992 10621 raft_consensus.cc:493] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.326725 10621 raft_consensus.cc:515] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.329774 10621 leader_election.cc:290] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:17.331691 9341 tablet_bootstrap.cc:492] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:17.332582 9341 ts_tablet_manager.cc:1403] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.207s user 0.012s sys 0.000s
I20260926 01:56:17.333480 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6e8059b94564c42894c6c8a3d695950" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:17.334122 8905 raft_consensus.cc:3060] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.335107 9341 raft_consensus.cc:359] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.335726 9341 raft_consensus.cc:385] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.335975 9341 raft_consensus.cc:740] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:17.336549 9341 consensus_queue.cc:260] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.344985 8901 raft_consensus.cc:3060] T 8711788a85f94c599224123f6b8cc814 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.356534 8843 leader_election.cc:304] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.357456 10800 raft_consensus.cc:2804] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:17.357875 10800 raft_consensus.cc:493] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:17.358268 10800 raft_consensus.cc:3060] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.368369 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c7151ffe5d94a0599bbe3e6c8999371" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:17.375939 9340 tablet_bootstrap.cc:654] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:17.354283 10684 raft_consensus.cc:493] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.395022 9341 ts_tablet_manager.cc:1434] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20260926 01:56:17.359382 9339 tablet_bootstrap.cc:492] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:17.421926 9339 ts_tablet_manager.cc:1403] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.324s user 0.005s sys 0.007s
I20260926 01:56:17.422348 10684 raft_consensus.cc:515] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.424636 10684 leader_election.cc:290] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:17.390959 9006 leader_election.cc:343] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.424751 9339 raft_consensus.cc:359] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.425377 9339 raft_consensus.cc:385] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.425630 9339 raft_consensus.cc:740] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:17.427366 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:17.427423 9339 consensus_queue.cc:260] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.378813 8905 raft_consensus.cc:2468] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:17.384665 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c7151ffe5d94a0599bbe3e6c8999371" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:17.431102 9339 ts_tablet_manager.cc:1434] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260926 01:56:17.431412 8897 raft_consensus.cc:2468] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:17.432283 9339 tablet_bootstrap.cc:492] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:17.386596 8843 leader_election.cc:304] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.432992 10731 raft_consensus.cc:2804] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:17.433364 10731 raft_consensus.cc:493] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:17.433671 10731 raft_consensus.cc:3060] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:17.438905 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 58967.9us
I20260926 01:56:17.445334 8843 leader_election.cc:304] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.444124 10731 raft_consensus.cc:515] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:17.430317 9006 leader_election.cc:343] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.460327 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cca4587d64bc4964b298ba1a74fe9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:17.462117 9341 tablet_bootstrap.cc:492] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:17.365530 10800 raft_consensus.cc:515] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.465708 10800 leader_election.cc:290] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:17.470762 9003 leader_election.cc:304] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.471963 10684 raft_consensus.cc:2804] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:17.472663 10684 raft_consensus.cc:697] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:17.473583 10684 consensus_queue.cc:237] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.447772 10731 leader_election.cc:290] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:17.489909 10805 raft_consensus.cc:2804] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:17.522642 10805 raft_consensus.cc:697] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:17.524277 10805 consensus_queue.cc:237] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.534060 9340 tablet_bootstrap.cc:492] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:17.534610 9340 ts_tablet_manager.cc:1403] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.297s user 0.009s sys 0.002s
I20260926 01:56:17.537130 9340 raft_consensus.cc:359] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.537797 9340 raft_consensus.cc:385] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.538034 9340 raft_consensus.cc:740] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:17.541337 9341 tablet_bootstrap.cc:654] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:17.579538 9340 consensus_queue.cc:260] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.581920 9340 ts_tablet_manager.cc:1434] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.047s user 0.004s sys 0.000s
W20260926 01:56:17.591885 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 121283us
W20260926 01:56:17.618944 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.507s (SENT)
I20260926 01:56:17.619580 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:17.645210 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:17.654425 9340 tablet_bootstrap.cc:492] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:17.659921 9341 tablet_bootstrap.cc:492] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:17.660521 9341 ts_tablet_manager.cc:1403] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.199s user 0.011s sys 0.002s
I20260926 01:56:17.663143 9341 raft_consensus.cc:359] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.668985 10748 raft_consensus.cc:493] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.669483 10748 raft_consensus.cc:515] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.671712 10748 leader_election.cc:290] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:17.673446 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec749c375d784786ba93020996708bb1" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:17.673980 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 80996.2us
I20260926 01:56:17.675076 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec749c375d784786ba93020996708bb1" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:17.686939 10816 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:17.695451 9066 raft_consensus.cc:2393] T ec749c375d784786ba93020996708bb1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:17.701159 9339 tablet_bootstrap.cc:654] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:17.709877 8843 leader_election.cc:343] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:17.710260 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 91531us
I20260926 01:56:17.710386 9152 leader_election.cc:304] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
W20260926 01:56:17.710635 8843 leader_election.cc:343] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.711274 10748 raft_consensus.cc:3060] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.664683 9341 raft_consensus.cc:385] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.725924 9341 raft_consensus.cc:740] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:17.726218 10748 raft_consensus.cc:2749] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:17.726733 9341 consensus_queue.cc:260] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.728302 9341 ts_tablet_manager.cc:1434] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.067s user 0.005s sys 0.000s
I20260926 01:56:17.732005 10762 raft_consensus.cc:493] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.732918 10762 raft_consensus.cc:515] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.735065 10762 leader_election.cc:290] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:17.735409 9340 tablet_bootstrap.cc:654] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:17.738359 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db85241141f4a54a2b7dce15412a6f9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:17.740638 8843 leader_election.cc:343] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.741860 10814 consensus_queue.cc:1048] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:56:17.748582 9003 leader_election.cc:343] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.751196 9003 leader_election.cc:304] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.752866 10684 raft_consensus.cc:2749] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:17.754103 9003 leader_election.cc:304] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.754705 10748 raft_consensus.cc:493] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.756999 10748 raft_consensus.cc:515] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.759990 10684 raft_consensus.cc:2804] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:17.760367 10684 raft_consensus.cc:493] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:17.760691 10684 raft_consensus.cc:3060] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.760866 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:17.761620 8905 raft_consensus.cc:2393] T 89fb965257de4db6b043b4a6dbe22709 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:17.762650 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89fb965257de4db6b043b4a6dbe22709" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:17.764866 9155 leader_election.cc:304] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:17.765656 10635 raft_consensus.cc:3060] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.768704 10684 raft_consensus.cc:515] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:17.769027 10748 leader_election.cc:290] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:17.771462 10684 leader_election.cc:290] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:17.772822 10635 raft_consensus.cc:2749] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:17.774870 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5c66728bb54ec6beb4f0a2c25d9105" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:17.778769 9339 tablet_bootstrap.cc:492] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:17.779284 9339 ts_tablet_manager.cc:1403] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.347s user 0.016s sys 0.002s
I20260926 01:56:17.784883 9339 raft_consensus.cc:359] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.790294 9339 raft_consensus.cc:385] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.792089 9339 raft_consensus.cc:740] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
W20260926 01:56:17.791718 9006 leader_election.cc:343] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.789012 9003 leader_election.cc:304] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.793560 10781 raft_consensus.cc:2804] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:17.793984 10781 raft_consensus.cc:697] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:17.794893 10781 consensus_queue.cc:237] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:17.806851 9340 tablet_bootstrap.cc:492] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:17.807334 9340 ts_tablet_manager.cc:1403] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.153s user 0.011s sys 0.000s
I20260926 01:56:17.808224 9341 tablet_bootstrap.cc:492] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:17.816388 10710 consensus_queue.cc:1048] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:17.819155 10710 consensus_queue.cc:1048] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:17.823547 9340 raft_consensus.cc:359] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.824260 9340 raft_consensus.cc:385] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:17.824512 9340 raft_consensus.cc:740] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:17.825227 9340 consensus_queue.cc:260] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.825502 9339 consensus_queue.cc:260] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.833743 9003 leader_election.cc:304] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.839071 10814 raft_consensus.cc:2804] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:17.839421 10814 raft_consensus.cc:493] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:17.839749 10814 raft_consensus.cc:3060] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.850358 10685 raft_consensus.cc:493] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:17.850916 10685 raft_consensus.cc:515] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.852881 10685 leader_election.cc:290] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:17.856596 9341 tablet_bootstrap.cc:654] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:17.858980 10814 raft_consensus.cc:515] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:17.864713 9340 ts_tablet_manager.cc:1434] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20260926 01:56:17.866988 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:17.860208 9003 leader_election.cc:304] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:17.836278 9339 ts_tablet_manager.cc:1434] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20260926 01:56:17.876547 10781 raft_consensus.cc:2804] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:17.877285 10781 raft_consensus.cc:697] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:17.878361 10781 consensus_queue.cc:237] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:17.892310 9006 leader_election.cc:343] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:17.894891 9339 tablet_bootstrap.cc:492] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:17.909336 10731 consensus_queue.cc:1048] T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:17.912595 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:17.935544 9340 tablet_bootstrap.cc:492] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:17.946828 9340 tablet_bootstrap.cc:654] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:17.973438 10814 leader_election.cc:290] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
I20260926 01:56:17.966588 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5c66728bb54ec6beb4f0a2c25d9105" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:17.974282 8905 raft_consensus.cc:3060] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:17.978425 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c7151ffe5d94a0599bbe3e6c8999371" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:17.979928 9006 leader_election.cc:343] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.009721 9339 tablet_bootstrap.cc:654] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:18.035398 9339 tablet_bootstrap.cc:492] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:18.035881 9339 ts_tablet_manager.cc:1403] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.141s user 0.010s sys 0.002s
I20260926 01:56:18.038743 8902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:18.041873 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cca4587d64bc4964b298ba1a74fe9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:18.042518 9066 raft_consensus.cc:3060] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.043233 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:18.043778 9069 raft_consensus.cc:3060] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.046983 10748 raft_consensus.cc:493] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.047483 10748 raft_consensus.cc:515] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.052052 10748 leader_election.cc:290] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:18.056381 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:18.057181 8900 raft_consensus.cc:2393] T e7512e1d08e74ef6a2c699352347eb8d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:18.057627 9069 raft_consensus.cc:2468] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:18.058171 9066 raft_consensus.cc:2468] T 504cca4587d64bc4964b298ba1a74fe9 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:18.059062 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7512e1d08e74ef6a2c699352347eb8d" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:18.060750 9155 consensus_peers.cc:597] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e12827ad93c47cf9013e238215ceb7a. This is attempt 6: this message will repeat every 5th retry.
I20260926 01:56:18.068476 9339 raft_consensus.cc:359] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.069173 9339 raft_consensus.cc:385] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.069435 9339 raft_consensus.cc:740] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.080075 9152 leader_election.cc:304] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:18.081300 10720 raft_consensus.cc:3060] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.086745 9339 consensus_queue.cc:260] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.089164 9339 ts_tablet_manager.cc:1434] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.053s user 0.002s sys 0.004s
I20260926 01:56:18.089763 10720 raft_consensus.cc:2749] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:18.090199 9339 tablet_bootstrap.cc:492] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:18.113340 8905 raft_consensus.cc:2468] T 1c5c66728bb54ec6beb4f0a2c25d9105 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:18.114687 9339 tablet_bootstrap.cc:654] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:18.124461 9341 tablet_bootstrap.cc:492] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
W20260926 01:56:18.127445 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 47445.9us
I20260926 01:56:18.135176 9339 tablet_bootstrap.cc:492] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:18.135614 9339 ts_tablet_manager.cc:1403] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.008s
I20260926 01:56:18.135280 9341 ts_tablet_manager.cc:1403] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.406s user 0.012s sys 0.000s
I20260926 01:56:18.138758 9339 raft_consensus.cc:359] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.139431 9339 raft_consensus.cc:385] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.142747 9339 raft_consensus.cc:740] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.143430 9339 consensus_queue.cc:260] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.152500 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db85241141f4a54a2b7dce15412a6f9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:18.153184 9066 raft_consensus.cc:2468] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:18.173974 9339 ts_tablet_manager.cc:1434] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20260926 01:56:18.191295 9339 tablet_bootstrap.cc:492] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:18.197754 9340 tablet_bootstrap.cc:492] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:18.198249 9340 ts_tablet_manager.cc:1403] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.263s user 0.008s sys 0.002s
I20260926 01:56:18.198619 9341 raft_consensus.cc:359] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.199338 9341 raft_consensus.cc:385] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.199602 9341 raft_consensus.cc:740] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.200968 9340 raft_consensus.cc:359] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.201629 9340 raft_consensus.cc:385] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.201894 9340 raft_consensus.cc:740] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.228184 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c7151ffe5d94a0599bbe3e6c8999371" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:18.228945 8905 raft_consensus.cc:3060] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.236063 10823 raft_consensus.cc:493] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.236562 10823 raft_consensus.cc:515] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.239903 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:18.241286 9006 leader_election.cc:343] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.239470 10823 leader_election.cc:290] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:18.234843 9339 tablet_bootstrap.cc:654] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:18.259822 9340 consensus_queue.cc:260] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.262143 9340 ts_tablet_manager.cc:1434] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.064s user 0.003s sys 0.001s
I20260926 01:56:18.266350 8843 leader_election.cc:304] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.267565 10738 raft_consensus.cc:2804] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:18.268025 10738 raft_consensus.cc:697] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:18.268867 10738 consensus_queue.cc:237] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.280223 9341 consensus_queue.cc:260] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.290073 10720 raft_consensus.cc:493] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.290591 10720 raft_consensus.cc:515] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.291188 8843 leader_election.cc:304] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.291992 10731 raft_consensus.cc:2804] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:18.292466 10731 raft_consensus.cc:697] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:18.293286 10731 consensus_queue.cc:237] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.294063 10720 leader_election.cc:290] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:18.294200 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6e8059b94564c42894c6c8a3d695950" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:18.307360 8843 consensus_peers.cc:597] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d56f3a5dfed4ac78f1da92e22db558a. This is attempt 6: this message will repeat every 5th retry.
I20260926 01:56:18.309041 9341 ts_tablet_manager.cc:1434] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.173s user 0.009s sys 0.000s
I20260926 01:56:18.311544 8900 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6e8059b94564c42894c6c8a3d695950" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:18.312330 8900 raft_consensus.cc:2393] T a6e8059b94564c42894c6c8a3d695950 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:18.315425 8843 leader_election.cc:304] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.317359 9340 tablet_bootstrap.cc:492] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:18.318333 9152 leader_election.cc:304] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:18.319244 10748 raft_consensus.cc:3060] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.321478 10738 raft_consensus.cc:2804] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:18.321914 10738 raft_consensus.cc:493] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:18.322229 10738 raft_consensus.cc:3060] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.329493 10748 raft_consensus.cc:2749] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:18.335248 9341 tablet_bootstrap.cc:492] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:18.341101 9339 tablet_bootstrap.cc:492] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:18.341586 9339 ts_tablet_manager.cc:1403] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.151s user 0.001s sys 0.012s
I20260926 01:56:18.310633 8905 raft_consensus.cc:2468] T 3c7151ffe5d94a0599bbe3e6c8999371 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:18.380878 9339 raft_consensus.cc:359] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.381621 9339 raft_consensus.cc:385] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.381914 9339 raft_consensus.cc:740] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.382839 9339 consensus_queue.cc:260] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.400416 9339 ts_tablet_manager.cc:1434] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.059s user 0.004s sys 0.004s
I20260926 01:56:18.413794 9339 tablet_bootstrap.cc:492] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:18.435926 10843 log.cc:927] Time spent T 2c0cd3e30f084966b27921e658cacf58 P 49bd320bf73f4252804ec867ec077510: Append to log took a long time: real 0.086s user 0.000s sys 0.002s
W20260926 01:56:18.442607 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 112221us
W20260926 01:56:18.448468 9003 leader_election.cc:343] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.448974 9003 leader_election.cc:304] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.449753 10824 raft_consensus.cc:2749] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:18.464155 10841 raft_consensus.cc:493] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.464883 10841 raft_consensus.cc:515] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.467842 10841 leader_election.cc:290] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:18.477573 9339 tablet_bootstrap.cc:654] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:18.374332 10738 raft_consensus.cc:515] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.486658 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:18.514127 9003 leader_election.cc:343] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.517234 9340 tablet_bootstrap.cc:654] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:18.519196 9003 leader_election.cc:304] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:56:18.489877 8843 leader_election.cc:343] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.521347 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db85241141f4a54a2b7dce15412a6f9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:18.521625 9339 tablet_bootstrap.cc:492] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:18.522060 9339 ts_tablet_manager.cc:1403] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.002s
I20260926 01:56:18.523803 9341 tablet_bootstrap.cc:654] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:18.523919 10684 raft_consensus.cc:2749] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:18.540009 8843 leader_election.cc:343] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.541662 10738 leader_election.cc:290] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:18.541447 9339 raft_consensus.cc:359] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.542171 9339 raft_consensus.cc:385] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.542443 9339 raft_consensus.cc:740] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.543120 9339 consensus_queue.cc:260] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.546772 9003 leader_election.cc:304] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.571738 10738 raft_consensus.cc:493] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.572309 10738 raft_consensus.cc:515] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.574551 10738 leader_election.cc:290] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:18.610610 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 63799us
I20260926 01:56:18.614375 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:18.621701 8843 leader_election.cc:343] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.625996 10738 consensus_queue.cc:1048] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:18.631902 9340 tablet_bootstrap.cc:492] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:18.636059 9339 ts_tablet_manager.cc:1434] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.114s user 0.009s sys 0.000s
I20260926 01:56:18.638761 9340 ts_tablet_manager.cc:1403] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.322s user 0.010s sys 0.005s
I20260926 01:56:18.649075 9341 tablet_bootstrap.cc:492] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:18.649808 9341 ts_tablet_manager.cc:1403] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.315s user 0.011s sys 0.000s
I20260926 01:56:18.653297 9341 raft_consensus.cc:359] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:18.654251 9341 raft_consensus.cc:385] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.654592 9341 raft_consensus.cc:740] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.613696 10814 raft_consensus.cc:2804] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:18.665100 10814 raft_consensus.cc:697] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:18.666141 10814 consensus_queue.cc:237] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:18.670354 10748 raft_consensus.cc:493] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.671065 10748 raft_consensus.cc:515] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.674043 10748 leader_election.cc:290] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:18.676447 9339 tablet_bootstrap.cc:492] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:18.676584 9341 consensus_queue.cc:260] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:18.680203 9341 ts_tablet_manager.cc:1434] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.030s user 0.005s sys 0.005s
I20260926 01:56:18.681631 9341 tablet_bootstrap.cc:492] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:18.690667 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 90228.6us
I20260926 01:56:18.695940 10684 raft_consensus.cc:493] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.696599 10684 raft_consensus.cc:515] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.696918 9340 raft_consensus.cc:359] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.697629 9340 raft_consensus.cc:385] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.697916 9340 raft_consensus.cc:740] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.698623 9340 consensus_queue.cc:260] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.699769 10684 leader_election.cc:290] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:18.709370 9339 tablet_bootstrap.cc:654] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:18.713572 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 101662us
I20260926 01:56:18.717857 9003 leader_election.cc:304] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.718624 10781 raft_consensus.cc:2804] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:18.719226 10781 raft_consensus.cc:697] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:18.720047 10781 consensus_queue.cc:237] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.723448 10850 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:18.724105 10850 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:18.725412 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:18.726029 8906 raft_consensus.cc:2468] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:18.728693 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:18.730758 10851 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:18.732025 10851 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:18.735013 10851 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:18.748481 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:18.750905 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:18.751521 9067 raft_consensus.cc:2468] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:18.759936 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:18.761786 10850 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:18.762506 10859 raft_consensus.cc:493] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.763067 10859 raft_consensus.cc:515] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.765698 10859 leader_election.cc:290] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:18.771385 9340 ts_tablet_manager.cc:1434] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.125s user 0.006s sys 0.002s
I20260926 01:56:18.775357 9339 tablet_bootstrap.cc:492] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:18.775918 9339 ts_tablet_manager.cc:1403] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.005s
I20260926 01:56:18.788956 9339 raft_consensus.cc:359] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.795425 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:18.797199 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c0cd3e30f084966b27921e658cacf58" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:18.790258 9339 raft_consensus.cc:385] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:18.809793 9339 raft_consensus.cc:740] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:18.810442 9339 consensus_queue.cc:260] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.812533 9339 ts_tablet_manager.cc:1434] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260926 01:56:18.814841 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db85241141f4a54a2b7dce15412a6f9" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:18.815380 9068 raft_consensus.cc:3060] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:18.816009 8526 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 01052a5e22a74a19919bae9dd0296291 (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 01052a5e22a74a19919bae9dd0296291 consensus state has no leader
I20260926 01:56:18.822146 9068 raft_consensus.cc:2468] T 9db85241141f4a54a2b7dce15412a6f9 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:18.825744 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:18.826357 9072 raft_consensus.cc:2468] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:18.832715 9339 tablet_bootstrap.cc:492] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:18.837563 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2156ef6b24c04ea38181832e7f18ff49" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:18.841632 9339 tablet_bootstrap.cc:654] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:18.875190 9155 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:18.880007 8843 leader_election.cc:304] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.881126 10738 raft_consensus.cc:2804] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:18.881484 10738 raft_consensus.cc:493] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:18.881783 10738 raft_consensus.cc:3060] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.884298 10859 raft_consensus.cc:493] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:18.884779 10859 raft_consensus.cc:515] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.887408 10859 leader_election.cc:290] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:18.891314 9152 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 53006.4us
I20260926 01:56:18.896047 9341 tablet_bootstrap.cc:654] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:18.913404 9006 consensus_peers.cc:597] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:18.914553 9006 leader_election.cc:343] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.915810 10859 consensus_queue.cc:1048] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:18.916296 8843 leader_election.cc:304] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.917483 10731 raft_consensus.cc:2804] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:18.917938 10731 raft_consensus.cc:697] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:18.918802 10731 consensus_queue.cc:237] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:18.919369 9006 leader_election.cc:343] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:18.926355 9340 tablet_bootstrap.cc:492] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:18.932674 9340 tablet_bootstrap.cc:654] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:18.935782 10762 consensus_queue.cc:1048] T ec749c375d784786ba93020996708bb1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:18.937986 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2156ef6b24c04ea38181832e7f18ff49" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:18.942778 8843 leader_election.cc:304] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.944044 10841 raft_consensus.cc:2804] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:18.944424 10841 raft_consensus.cc:493] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:18.944748 10841 raft_consensus.cc:3060] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.945259 9152 leader_election.cc:304] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:18.946120 10720 raft_consensus.cc:2749] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:18.949324 9152 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:18.949719 9152 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:18.963769 10841 raft_consensus.cc:515] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:18.966918 10720 raft_consensus.cc:2749] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:18.968729 10841 leader_election.cc:290] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:18.972227 9003 leader_election.cc:304] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:18.985580 10859 raft_consensus.cc:2804] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:18.986147 10859 raft_consensus.cc:493] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:18.986661 10859 raft_consensus.cc:3060] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:18.988521 9339 tablet_bootstrap.cc:492] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:18.992596 9339 ts_tablet_manager.cc:1403] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.160s user 0.020s sys 0.010s
I20260926 01:56:19.003264 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:19.008883 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 59590.4us
I20260926 01:56:19.020951 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:18.973845 9340 tablet_bootstrap.cc:492] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:19.030856 9340 ts_tablet_manager.cc:1403] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.000s
I20260926 01:56:19.033629 9340 raft_consensus.cc:359] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.034173 10868 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:19.034312 9340 raft_consensus.cc:385] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.035742 9340 raft_consensus.cc:740] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.036864 9340 consensus_queue.cc:260] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.042827 9340 ts_tablet_manager.cc:1434] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260926 01:56:19.034684 10868 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:19.049677 9340 tablet_bootstrap.cc:492] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:18.998643 10738 raft_consensus.cc:515] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:19.005321 8843 leader_election.cc:343] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:19.067327 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 62044.9us
I20260926 01:56:19.075230 10738 leader_election.cc:290] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:19.079244 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:19.081269 8843 leader_election.cc:343] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.084195 9339 raft_consensus.cc:359] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.084856 9339 raft_consensus.cc:385] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.085099 9339 raft_consensus.cc:740] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.085696 9339 consensus_queue.cc:260] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:19.087045 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 70142us
W20260926 01:56:19.088521 9006 leader_election.cc:343] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.089268 10856 raft_consensus.cc:493] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.089660 10856 raft_consensus.cc:515] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.094821 10859 raft_consensus.cc:515] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.096107 10856 leader_election.cc:290] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:19.096779 10859 leader_election.cc:290] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:19.101418 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f300124bc3a4177a606b116bfef3a65" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:19.102162 9339 ts_tablet_manager.cc:1434] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.109s user 0.004s sys 0.002s
I20260926 01:56:19.102787 9215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:19.103209 9339 tablet_bootstrap.cc:492] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:19.107025 9006 leader_election.cc:343] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:19.108318 9006 leader_election.cc:343] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:19.114111 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 81097.5us
W20260926 01:56:19.115756 9003 leader_election.cc:343] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.116137 9003 leader_election.cc:304] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.117311 10823 raft_consensus.cc:2749] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:19.121033 9340 tablet_bootstrap.cc:654] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:19.124662 9339 tablet_bootstrap.cc:654] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:19.136118 9341 tablet_bootstrap.cc:492] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:19.136620 9341 ts_tablet_manager.cc:1403] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.455s user 0.012s sys 0.000s
I20260926 01:56:19.151070 9339 tablet_bootstrap.cc:492] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:19.151662 9339 ts_tablet_manager.cc:1403] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.002s
I20260926 01:56:19.159255 9339 raft_consensus.cc:359] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.159976 9339 raft_consensus.cc:385] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.160243 9339 raft_consensus.cc:740] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.189175 9219 raft_consensus.cc:3060] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:19.206435 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 88676.4us
I20260926 01:56:19.207598 10855 consensus_queue.cc:1048] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:19.208051 10781 consensus_queue.cc:1048] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:19.211006 9003 leader_election.cc:343] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.229470 9341 raft_consensus.cc:359] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:19.230271 9341 raft_consensus.cc:385] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.230563 9341 raft_consensus.cc:740] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.231312 9341 consensus_queue.cc:260] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:19.246197 10868 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:19.247537 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:19.252172 10851 raft_consensus.cc:493] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:19.252751 10851 raft_consensus.cc:515] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:19.255141 10851 leader_election.cc:290] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:19.257023 9155 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
W20260926 01:56:19.257494 9152 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:19.257874 9152 leader_election.cc:304] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:19.259461 10720 raft_consensus.cc:2749] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:19.174391 9339 consensus_queue.cc:260] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.263549 9339 ts_tablet_manager.cc:1434] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.107s user 0.007s sys 0.002s
I20260926 01:56:19.264657 9339 tablet_bootstrap.cc:492] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:19.256301 9003 leader_election.cc:304] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.279678 10855 raft_consensus.cc:2749] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:19.280738 10738 raft_consensus.cc:493] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.281266 10738 raft_consensus.cc:515] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.283528 10738 leader_election.cc:290] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:19.286787 9341 ts_tablet_manager.cc:1434] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.150s user 0.009s sys 0.000s
W20260926 01:56:19.279853 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 69180.2us
I20260926 01:56:19.294782 9215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:19.299077 8843 leader_election.cc:343] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.307013 9340 tablet_bootstrap.cc:492] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:19.307497 9340 ts_tablet_manager.cc:1403] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.258s user 0.013s sys 0.005s
I20260926 01:56:19.310714 9340 raft_consensus.cc:359] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.316092 9340 raft_consensus.cc:385] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.316848 9340 raft_consensus.cc:740] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.323398 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:19.340298 10855 raft_consensus.cc:493] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.340938 10855 raft_consensus.cc:515] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.341697 10854 raft_consensus.cc:493] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.342144 10854 raft_consensus.cc:515] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.344097 10854 leader_election.cc:290] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:19.344812 10855 leader_election.cc:290] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:19.347627 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:19.348687 9215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:19.350742 9006 leader_election.cc:343] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:19.351491 9006 leader_election.cc:343] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.353770 9341 tablet_bootstrap.cc:492] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:19.364537 10851 raft_consensus.cc:493] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.365151 10851 raft_consensus.cc:515] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:19.367362 10851 leader_election.cc:290] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:19.368626 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:19.368750 8907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6752df6a58914df5b8718dd0439aa885" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:19.369434 8907 raft_consensus.cc:2393] T 6752df6a58914df5b8718dd0439aa885 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:19.371618 9152 leader_election.cc:304] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:19.372749 10851 raft_consensus.cc:3060] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:19.378816 8843 consensus_peers.cc:597] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:19.318428 9340 consensus_queue.cc:260] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.400264 9340 ts_tablet_manager.cc:1434] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.092s user 0.004s sys 0.003s
I20260926 01:56:19.417362 9339 tablet_bootstrap.cc:654] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:19.419225 10851 raft_consensus.cc:2749] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:19.424654 8906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f300124bc3a4177a606b116bfef3a65" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:19.425271 8906 raft_consensus.cc:2468] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
I20260926 01:56:19.428150 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
W20260926 01:56:19.428428 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 53256us
I20260926 01:56:19.428892 8905 raft_consensus.cc:2393] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
W20260926 01:56:19.463003 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 50165.8us
W20260926 01:56:19.491142 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 47468.1us
I20260926 01:56:19.497951 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:19.498637 9067 raft_consensus.cc:3060] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:19.501543 9341 tablet_bootstrap.cc:654] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:19.508744 10851 raft_consensus.cc:493] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.509436 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:19.510260 9066 raft_consensus.cc:2393] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:19.511873 10851 raft_consensus.cc:515] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:19.515543 9065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:19.516336 9065 raft_consensus.cc:2468] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:19.440407 9340 tablet_bootstrap.cc:492] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:19.520846 10859 consensus_queue.cc:1048] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:19.536710 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b612e8955d5f42dd8658320031fe36ca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:19.537635 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b612e8955d5f42dd8658320031fe36ca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:19.538267 8843 consensus_peers.cc:597] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:19.539464 9155 leader_election.cc:304] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:19.536688 10851 leader_election.cc:290] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:19.543586 9067 raft_consensus.cc:2468] T 6629d125861f4e27b61c10d585a35c10 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:19.547588 10748 raft_consensus.cc:2749] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:19.557449 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:19.558645 9339 tablet_bootstrap.cc:492] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:19.559140 9339 ts_tablet_manager.cc:1403] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.295s user 0.011s sys 0.002s
I20260926 01:56:19.575802 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:19.583523 9216 raft_consensus.cc:3060] T b612e8955d5f42dd8658320031fe36ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:19.609297 9341 tablet_bootstrap.cc:492] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:19.609794 9341 ts_tablet_manager.cc:1403] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.256s user 0.005s sys 0.009s
I20260926 01:56:19.561753 9339 raft_consensus.cc:359] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.616879 9339 raft_consensus.cc:385] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.617205 9339 raft_consensus.cc:740] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.643864 9340 tablet_bootstrap.cc:654] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:19.649362 10855 consensus_queue.cc:1048] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:19.617911 9339 consensus_queue.cc:260] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:19.653417 9006 consensus_peers.cc:597] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:19.654073 9339 ts_tablet_manager.cc:1434] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.095s user 0.006s sys 0.000s
I20260926 01:56:19.657688 8843 leader_election.cc:304] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.659408 10731 raft_consensus.cc:2804] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:19.659781 10731 raft_consensus.cc:493] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:19.660100 10731 raft_consensus.cc:3060] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:19.669710 10731 raft_consensus.cc:515] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.678658 10841 consensus_queue.cc:1048] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:19.674794 9341 raft_consensus.cc:359] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.699656 9341 raft_consensus.cc:385] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.700538 9341 raft_consensus.cc:740] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.702255 9341 consensus_queue.cc:260] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.681650 8843 leader_election.cc:304] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:19.707263 10841 raft_consensus.cc:2749] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:56:19.708951 9339 tablet_bootstrap.cc:492] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:19.713068 10731 leader_election.cc:290] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:19.715567 9003 leader_election.cc:343] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.716851 9003 leader_election.cc:304] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.717583 10823 raft_consensus.cc:2749] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:19.718541 10841 consensus_queue.cc:1048] T 504cca4587d64bc4964b298ba1a74fe9 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:19.736069 9340 tablet_bootstrap.cc:492] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:19.737419 9340 ts_tablet_manager.cc:1403] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.299s user 0.013s sys 0.003s
I20260926 01:56:19.747449 8843 leader_election.cc:304] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.751042 10731 raft_consensus.cc:2804] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:19.751636 10731 raft_consensus.cc:697] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:19.753190 9341 ts_tablet_manager.cc:1434] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.143s user 0.006s sys 0.005s
I20260926 01:56:19.752661 10731 consensus_queue.cc:237] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.789196 9003 leader_election.cc:304] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.794953 9003 leader_election.cc:304] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.796120 10855 raft_consensus.cc:2749] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260926 01:56:19.807325 9910 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.600s (SENT) [suppressed 2 similar messages]
I20260926 01:56:19.829787 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:19.836648 10738 raft_consensus.cc:493] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.837165 10738 raft_consensus.cc:515] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.839244 10738 leader_election.cc:290] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:19.830279 9341 tablet_bootstrap.cc:492] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:19.845336 9339 tablet_bootstrap.cc:654] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:19.799206 10854 raft_consensus.cc:2804] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:19.872736 10854 raft_consensus.cc:493] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:19.873148 10854 raft_consensus.cc:3060] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:19.882495 9340 raft_consensus.cc:359] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.883342 9340 raft_consensus.cc:385] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:19.883620 9340 raft_consensus.cc:740] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:19.884333 9340 consensus_queue.cc:260] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.885901 9340 ts_tablet_manager.cc:1434] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.148s user 0.001s sys 0.006s
I20260926 01:56:19.885440 10781 raft_consensus.cc:493] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.897236 10781 raft_consensus.cc:515] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:19.896876 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 89477.2us
I20260926 01:56:19.899524 10781 leader_election.cc:290] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:19.909005 10854 raft_consensus.cc:515] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.918370 10854 leader_election.cc:290] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:19.930564 10851 raft_consensus.cc:493] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:19.931195 10851 raft_consensus.cc:515] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:19.933473 10851 leader_election.cc:290] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:19.934688 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:19.942075 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 147197us
I20260926 01:56:19.955569 9069 raft_consensus.cc:2393] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
W20260926 01:56:19.956107 9003 leader_election.cc:343] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.956517 9003 leader_election.cc:304] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.958513 10852 raft_consensus.cc:2749] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:19.958014 9003 leader_election.cc:343] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:19.959833 9003 leader_election.cc:304] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:19.960458 10855 raft_consensus.cc:2749] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:19.977424 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812fe778734d4210b744ed8e45d412f1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:19.979386 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:19.980118 9215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f300124bc3a4177a606b116bfef3a65" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:19.984097 9152 leader_election.cc:304] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
W20260926 01:56:20.004163 9006 consensus_peers.cc:597] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:20.010249 10851 raft_consensus.cc:3060] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:20.022935 9006 leader_election.cc:343] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.015148 9339 tablet_bootstrap.cc:492] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:20.030968 9339 ts_tablet_manager.cc:1403] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.350s user 0.009s sys 0.004s
W20260926 01:56:20.033245 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 113146us
W20260926 01:56:20.037626 8843 leader_election.cc:343] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.045492 9339 raft_consensus.cc:359] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.046393 9339 raft_consensus.cc:385] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:20.046770 9339 raft_consensus.cc:740] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
W20260926 01:56:20.047025 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.617s (SENT)
I20260926 01:56:20.047613 9339 consensus_queue.cc:260] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.049316 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:20.063871 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:20.087471 10898 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:20.093214 10851 raft_consensus.cc:2749] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:20.110576 9339 ts_tablet_manager.cc:1434] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.079s user 0.006s sys 0.000s
I20260926 01:56:20.111634 9339 tablet_bootstrap.cc:492] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:20.124758 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 63844.1us
W20260926 01:56:20.131652 9006 leader_election.cc:343] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.085817 9340 tablet_bootstrap.cc:492] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:20.097937 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812fe778734d4210b744ed8e45d412f1" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:20.099509 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f300124bc3a4177a606b116bfef3a65" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:20.140358 8899 raft_consensus.cc:3060] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:20.149606 8899 raft_consensus.cc:2468] T 5f300124bc3a4177a606b116bfef3a65 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:20.162809 9066 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:20.163493 9066 raft_consensus.cc:2468] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:20.161216 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:20.166522 9069 raft_consensus.cc:3060] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:20.183646 9069 raft_consensus.cc:2468] T e85d17e9e48e4e54956e5c8460f7c9a8 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:20.188531 9341 tablet_bootstrap.cc:654] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:20.230772 10731 raft_consensus.cc:493] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:20.231271 10731 raft_consensus.cc:515] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.233685 10731 leader_election.cc:290] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:20.222095 10781 consensus_queue.cc:1048] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:56:20.240767 9003 leader_election.cc:343] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.267823 9003 leader_election.cc:304] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:20.269747 10891 raft_consensus.cc:2749] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:20.271103 9003 leader_election.cc:304] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:20.272435 10823 raft_consensus.cc:2804] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:20.272935 10823 raft_consensus.cc:697] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [term 1 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:20.273823 10823 consensus_queue.cc:237] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.274521 10891 consensus_queue.cc:1048] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260926 01:56:20.282763 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 64634.8us
I20260926 01:56:20.320856 9341 tablet_bootstrap.cc:492] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:20.321446 9341 ts_tablet_manager.cc:1403] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.530s user 0.013s sys 0.004s
I20260926 01:56:20.324596 9341 raft_consensus.cc:359] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.325444 9340 tablet_bootstrap.cc:654] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:20.343870 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 45007.6us
I20260926 01:56:20.393167 10841 consensus_queue.cc:1048] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:20.396724 9339 tablet_bootstrap.cc:654] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:20.400427 8843 leader_election.cc:304] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:20.423660 10841 raft_consensus.cc:2804] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:20.424106 10841 raft_consensus.cc:493] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:20.424436 10841 raft_consensus.cc:3060] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:20.432250 10841 raft_consensus.cc:515] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.452828 9341 raft_consensus.cc:385] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:20.453169 9341 raft_consensus.cc:740] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:20.485430 10891 raft_consensus.cc:493] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:20.486135 10891 raft_consensus.cc:515] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.494242 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:20.495937 9006 leader_election.cc:343] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.498160 10891 leader_election.cc:290] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:20.516343 10841 leader_election.cc:290] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:20.453881 9341 consensus_queue.cc:260] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.522668 8843 leader_election.cc:304] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:20.547931 9341 ts_tablet_manager.cc:1434] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.226s user 0.001s sys 0.004s
I20260926 01:56:20.549612 9341 tablet_bootstrap.cc:492] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:20.574795 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.600s (SENT)
I20260926 01:56:20.575313 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:20.589740 9340 tablet_bootstrap.cc:492] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:20.590288 9340 ts_tablet_manager.cc:1403] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.505s user 0.010s sys 0.005s
W20260926 01:56:20.576452 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 52763.3us
I20260926 01:56:20.601598 10913 raft_consensus.cc:2804] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:20.648636 9339 tablet_bootstrap.cc:492] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:20.649147 9339 ts_tablet_manager.cc:1403] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.538s user 0.012s sys 0.000s
I20260926 01:56:20.650956 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:20.660233 10916 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260926 01:56:20.663731 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 88981.7us
I20260926 01:56:20.684885 9339 raft_consensus.cc:359] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.685614 9339 raft_consensus.cc:385] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:20.685885 9339 raft_consensus.cc:740] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:20.687884 9339 consensus_queue.cc:260] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.687477 9340 raft_consensus.cc:359] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.688869 9340 raft_consensus.cc:385] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:20.689189 9340 raft_consensus.cc:740] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:20.696967 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:20.712561 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:20.713286 9067 raft_consensus.cc:2468] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:20.721381 9340 consensus_queue.cc:260] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.708081 10913 raft_consensus.cc:697] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:20.722923 10913 consensus_queue.cc:237] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.745543 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:20.756665 9341 tablet_bootstrap.cc:654] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:20.767604 9340 ts_tablet_manager.cc:1434] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.177s user 0.005s sys 0.001s
I20260926 01:56:20.794859 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:20.795490 9067 raft_consensus.cc:3060] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:20.800132 9220 raft_consensus.cc:3060] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:20.804996 9003 leader_election.cc:343] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.805480 9003 leader_election.cc:304] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:20.807225 10891 raft_consensus.cc:493] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:20.807724 10891 raft_consensus.cc:515] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:20.809671 10891 leader_election.cc:290] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:20.811978 10891 raft_consensus.cc:2749] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:20.819590 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:20.823223 9340 tablet_bootstrap.cc:492] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:20.825203 9067 raft_consensus.cc:2468] T 094b7485380b46fe8e1fef23ac50b830 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
W20260926 01:56:20.826056 9006 leader_election.cc:343] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.830500 9339 ts_tablet_manager.cc:1434] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.181s user 0.002s sys 0.007s
I20260926 01:56:20.852949 9339 tablet_bootstrap.cc:492] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:20.884289 8843 leader_election.cc:343] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:20.903308 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:20.885862 10922 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
W20260926 01:56:20.904537 8843 consensus_peers.cc:597] T 9db85241141f4a54a2b7dce15412a6f9 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:20.912949 10922 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
W20260926 01:56:20.969430 8843 leader_election.cc:343] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:20.974141 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 88350.3us
I20260926 01:56:20.979009 9340 tablet_bootstrap.cc:654] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:20.984690 8843 leader_election.cc:343] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.014776 10893 raft_consensus.cc:493] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:21.015309 10893 raft_consensus.cc:515] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.018060 9341 tablet_bootstrap.cc:492] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:21.019241 10893 leader_election.cc:290] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:21.021386 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:21.022740 9006 leader_election.cc:343] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:21.031517 9003 leader_election.cc:343] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.032114 9003 leader_election.cc:304] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.032814 10852 raft_consensus.cc:2749] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:21.039040 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:21.038017 10922 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:21.042601 9339 tablet_bootstrap.cc:654] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.062193 9341 ts_tablet_manager.cc:1403] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.513s user 0.007s sys 0.007s
I20260926 01:56:21.065018 9341 raft_consensus.cc:359] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.087642 9341 raft_consensus.cc:385] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.087989 9341 raft_consensus.cc:740] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.098869 9341 consensus_queue.cc:260] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.104018 9341 ts_tablet_manager.cc:1434] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.041s user 0.002s sys 0.004s
I20260926 01:56:21.105077 9341 tablet_bootstrap.cc:492] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:21.124306 9339 tablet_bootstrap.cc:492] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:21.124889 9339 ts_tablet_manager.cc:1403] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.272s user 0.016s sys 0.000s
I20260926 01:56:21.128813 9339 raft_consensus.cc:359] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.129577 9339 raft_consensus.cc:385] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.129887 9339 raft_consensus.cc:740] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.132210 9341 tablet_bootstrap.cc:654] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.146189 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:21.149894 9340 tablet_bootstrap.cc:492] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:21.150478 9340 ts_tablet_manager.cc:1403] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.327s user 0.011s sys 0.004s
I20260926 01:56:21.153247 9340 raft_consensus.cc:359] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.154098 9340 raft_consensus.cc:385] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.154420 9340 raft_consensus.cc:740] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.155190 9340 consensus_queue.cc:260] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.159863 9340 ts_tablet_manager.cc:1434] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260926 01:56:21.161041 9340 tablet_bootstrap.cc:492] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:21.176882 9340 tablet_bootstrap.cc:654] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.198628 9340 tablet_bootstrap.cc:492] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:21.199200 9340 ts_tablet_manager.cc:1403] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.009s
I20260926 01:56:21.203331 9339 consensus_queue.cc:260] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.210091 10913 raft_consensus.cc:493] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:21.210587 10913 raft_consensus.cc:515] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.212502 10913 leader_election.cc:290] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:21.217496 9340 raft_consensus.cc:359] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.218707 9339 ts_tablet_manager.cc:1434] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.093s user 0.004s sys 0.000s
I20260926 01:56:21.218644 9340 raft_consensus.cc:385] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.219239 9340 raft_consensus.cc:740] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.220016 9340 consensus_queue.cc:260] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.228080 9340 ts_tablet_manager.cc:1434] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.029s user 0.004s sys 0.001s
I20260926 01:56:21.230782 9340 tablet_bootstrap.cc:492] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:21.237669 9340 tablet_bootstrap.cc:654] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.254308 9341 tablet_bootstrap.cc:492] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:21.254820 9341 ts_tablet_manager.cc:1403] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.150s user 0.010s sys 0.001s
I20260926 01:56:21.257332 9341 raft_consensus.cc:359] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:21.257998 9341 raft_consensus.cc:385] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.277515 9340 tablet_bootstrap.cc:492] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:21.278090 9340 ts_tablet_manager.cc:1403] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.012s
I20260926 01:56:21.280285 9339 tablet_bootstrap.cc:492] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:21.294713 8843 leader_election.cc:304] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.296129 8843 leader_election.cc:304] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.297326 10914 raft_consensus.cc:2804] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:21.297670 10914 raft_consensus.cc:493] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:21.297972 10914 raft_consensus.cc:3060] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.299706 10731 raft_consensus.cc:2804] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:21.300143 10731 raft_consensus.cc:697] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:21.300855 10731 consensus_queue.cc:237] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:21.315990 9003 leader_election.cc:343] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.316506 9003 leader_election.cc:304] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.322763 10781 raft_consensus.cc:2749] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:21.324645 10912 raft_consensus.cc:493] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:21.325207 10912 raft_consensus.cc:515] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.328811 10912 leader_election.cc:290] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:21.344166 10914 raft_consensus.cc:515] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.345947 10781 consensus_queue.cc:1048] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:21.258239 9341 raft_consensus.cc:740] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.356980 9341 consensus_queue.cc:260] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:21.359303 9341 ts_tablet_manager.cc:1434] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.104s user 0.006s sys 0.000s
I20260926 01:56:21.353179 9340 raft_consensus.cc:359] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:21.373705 8525 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 01052a5e22a74a19919bae9dd0296291 (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 01052a5e22a74a19919bae9dd0296291 consensus state has no leader
I20260926 01:56:21.366385 10914 leader_election.cc:290] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:21.371066 9340 raft_consensus.cc:385] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.377837 9340 raft_consensus.cc:740] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.378851 9340 consensus_queue.cc:260] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.406935 9340 ts_tablet_manager.cc:1434] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.128s user 0.003s sys 0.005s
I20260926 01:56:21.408090 9341 tablet_bootstrap.cc:492] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:21.411770 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 114198us
I20260926 01:56:21.417773 9340 tablet_bootstrap.cc:492] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:21.418558 10738 consensus_queue.cc:1048] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:21.419955 10738 raft_consensus.cc:493] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:21.420367 10738 raft_consensus.cc:515] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.422204 10738 leader_election.cc:290] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:21.429669 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:21.431881 9216 raft_consensus.cc:3060] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.449326 9339 tablet_bootstrap.cc:700] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2: Time spent opening tablet: real 0.164s user 0.003s sys 0.000s
I20260926 01:56:21.460764 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:21.461453 9072 raft_consensus.cc:2468] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:21.464331 9339 tablet_bootstrap.cc:654] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:21.464643 9006 consensus_peers.cc:597] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:21.463774 9215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:21.464052 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:21.463774 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812fe778734d4210b744ed8e45d412f1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:21.438038 9341 tablet_bootstrap.cc:654] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.494127 9339 tablet_bootstrap.cc:492] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:21.494697 9339 ts_tablet_manager.cc:1403] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.275s user 0.012s sys 0.000s
I20260926 01:56:21.498868 9340 tablet_bootstrap.cc:654] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.503778 9339 raft_consensus.cc:359] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.517532 9339 raft_consensus.cc:385] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.517886 9339 raft_consensus.cc:740] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.521204 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:21.521843 9072 raft_consensus.cc:2468] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:21.526917 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:21.527590 9072 raft_consensus.cc:3060] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.528316 9341 tablet_bootstrap.cc:492] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:21.528752 9341 ts_tablet_manager.cc:1403] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.000s
I20260926 01:56:21.531551 9341 raft_consensus.cc:359] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.532444 9341 raft_consensus.cc:385] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.532702 9341 raft_consensus.cc:740] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.533375 9341 consensus_queue.cc:260] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:21.537984 8843 consensus_peers.cc:597] T 6629d125861f4e27b61c10d585a35c10 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:21.538898 9072 raft_consensus.cc:2468] T 4efbc1e1ef35420db51742d99d5a32bf P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:21.540264 10914 raft_consensus.cc:493] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:21.540725 10914 raft_consensus.cc:515] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.546140 10914 leader_election.cc:290] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:21.556396 8843 leader_election.cc:343] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.539578 9339 consensus_queue.cc:260] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.549170 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812fe778734d4210b744ed8e45d412f1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:21.559139 9073 raft_consensus.cc:2468] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:21.561655 9339 ts_tablet_manager.cc:1434] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20260926 01:56:21.562693 9339 tablet_bootstrap.cc:492] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:21.584175 8843 leader_election.cc:343] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:21.584856 8843 leader_election.cc:343] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:21.585394 8843 leader_election.cc:343] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.586969 9341 ts_tablet_manager.cc:1434] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20260926 01:56:21.591590 9341 tablet_bootstrap.cc:492] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:21.596200 10851 raft_consensus.cc:493] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:21.596694 10851 raft_consensus.cc:515] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
W20260926 01:56:21.600455 9155 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
I20260926 01:56:21.601157 10851 leader_election.cc:290] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:21.603927 9152 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
I20260926 01:56:21.604517 9152 leader_election.cc:304] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:21.605235 10851 raft_consensus.cc:2749] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:21.621225 9218 raft_consensus.cc:3060] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.621999 9339 tablet_bootstrap.cc:654] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.626251 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:21.626910 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:21.627544 9072 raft_consensus.cc:2468] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
W20260926 01:56:21.635254 8599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.576s (SENT) [suppressed 2 similar messages]
W20260926 01:56:21.643838 8646 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac3e0 after lost signal to thread 8581
W20260926 01:56:21.644646 8646 debug-util.cc:398] Leaking SignalData structure 0x7b0800099e00 after lost signal to thread 8649
I20260926 01:56:21.655278 9340 tablet_bootstrap.cc:492] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:21.655779 9340 ts_tablet_manager.cc:1403] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.238s user 0.005s sys 0.008s
W20260926 01:56:21.661774 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 43631.5us
I20260926 01:56:21.671957 9340 raft_consensus.cc:359] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.672740 9340 raft_consensus.cc:385] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.673002 9340 raft_consensus.cc:740] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.673643 9340 consensus_queue.cc:260] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.664368 8843 leader_election.cc:304] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.682706 9340 ts_tablet_manager.cc:1434] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260926 01:56:21.683149 10942 raft_consensus.cc:2804] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:21.683471 8843 leader_election.cc:304] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.683604 10942 raft_consensus.cc:493] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:21.684386 10942 raft_consensus.cc:3060] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.683717 9340 tablet_bootstrap.cc:492] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:21.714246 9339 tablet_bootstrap.cc:492] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:21.715575 9339 ts_tablet_manager.cc:1403] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.153s user 0.007s sys 0.005s
I20260926 01:56:21.719239 9339 raft_consensus.cc:359] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.719915 9339 raft_consensus.cc:385] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.720196 9339 raft_consensus.cc:740] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.684554 8843 leader_election.cc:304] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.684846 10913 raft_consensus.cc:2804] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:21.723285 10913 raft_consensus.cc:493] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:21.723683 10913 raft_consensus.cc:3060] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.725649 10914 raft_consensus.cc:2804] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:21.726164 10914 raft_consensus.cc:697] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:21.727106 10914 consensus_queue.cc:237] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:21.742789 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 58300.1us
I20260926 01:56:21.743582 8843 leader_election.cc:304] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.736527 9340 tablet_bootstrap.cc:654] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.747599 10738 raft_consensus.cc:2804] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:21.747993 10738 raft_consensus.cc:493] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:21.748303 10738 raft_consensus.cc:3060] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.756453 10738 raft_consensus.cc:515] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.758176 10738 leader_election.cc:290] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:21.774299 10895 raft_consensus.cc:493] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:21.774960 10895 raft_consensus.cc:515] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.777743 10895 leader_election.cc:290] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:21.781705 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:21.814862 9003 leader_election.cc:343] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.821353 10893 raft_consensus.cc:493] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:21.821851 10893 raft_consensus.cc:515] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.823971 10893 leader_election.cc:290] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:21.782181 9339 consensus_queue.cc:260] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.826426 9339 ts_tablet_manager.cc:1434] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.111s user 0.006s sys 0.001s
I20260926 01:56:21.808110 9340 tablet_bootstrap.cc:492] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:21.803294 8843 leader_election.cc:304] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:21.842999 9340 ts_tablet_manager.cc:1403] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.159s user 0.013s sys 0.000s
I20260926 01:56:21.843307 10738 raft_consensus.cc:2804] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:21.843663 10738 raft_consensus.cc:493] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:21.843991 10738 raft_consensus.cc:3060] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.847244 9340 raft_consensus.cc:359] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.848371 9340 raft_consensus.cc:385] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:21.848642 9340 raft_consensus.cc:740] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:21.849326 9340 consensus_queue.cc:260] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.855737 9340 ts_tablet_manager.cc:1434] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260926 01:56:21.856997 9340 tablet_bootstrap.cc:492] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:21.860217 9339 tablet_bootstrap.cc:492] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:21.845474 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 43628us
I20260926 01:56:21.879201 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812fe778734d4210b744ed8e45d412f1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:21.880214 10738 raft_consensus.cc:515] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.880447 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812fe778734d4210b744ed8e45d412f1" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:21.881103 9071 raft_consensus.cc:3060] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.882048 10738 leader_election.cc:290] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:21.892342 8843 leader_election.cc:343] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:21.897198 8843 leader_election.cc:343] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:21.897915 8843 consensus_peers.cc:597] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:21.899834 9215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:21.903195 8843 leader_election.cc:343] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.903537 9071 raft_consensus.cc:2468] T 812fe778734d4210b744ed8e45d412f1 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:21.904539 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:21.905193 9069 raft_consensus.cc:3060] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:21.905587 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 270334us
I20260926 01:56:21.815027 10913 raft_consensus.cc:515] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.908881 10913 leader_election.cc:290] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:21.930902 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:21.924504 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:21.931614 9068 raft_consensus.cc:3060] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:21.932834 8843 leader_election.cc:304] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.932719 10941 consensus_queue.cc:1048] T e85d17e9e48e4e54956e5c8460f7c9a8 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:21.934093 10941 raft_consensus.cc:2804] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:21.934579 10941 raft_consensus.cc:697] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:21.935417 10941 consensus_queue.cc:237] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:21.942113 9339 tablet_bootstrap.cc:654] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.946048 9341 tablet_bootstrap.cc:654] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:21.948405 9340 tablet_bootstrap.cc:654] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:21.948558 8843 leader_election.cc:343] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:21.953138 9068 raft_consensus.cc:2468] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:21.896939 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:21.975083 8843 leader_election.cc:304] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:21.975867 10912 raft_consensus.cc:2804] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:21.976368 10912 raft_consensus.cc:697] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:21.977221 10912 consensus_queue.cc:237] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.001246 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:22.006037 9340 tablet_bootstrap.cc:492] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:22.006551 9340 ts_tablet_manager.cc:1403] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.150s user 0.012s sys 0.007s
I20260926 01:56:22.009289 9340 raft_consensus.cc:359] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.011751 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:22.023397 9339 tablet_bootstrap.cc:492] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:22.023877 9339 ts_tablet_manager.cc:1403] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.164s user 0.014s sys 0.002s
I20260926 01:56:22.000165 10942 raft_consensus.cc:515] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.027319 9339 raft_consensus.cc:359] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.027990 9339 raft_consensus.cc:385] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.028251 9339 raft_consensus.cc:740] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.029022 10942 leader_election.cc:290] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.029682 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:22.031889 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:22.032694 9071 raft_consensus.cc:3060] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:22.009964 9340 raft_consensus.cc:385] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.035461 9340 raft_consensus.cc:740] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.036219 9340 consensus_queue.cc:260] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.037490 9339 consensus_queue.cc:260] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.039218 9339 ts_tablet_manager.cc:1434] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
W20260926 01:56:22.041575 8843 leader_election.cc:343] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.056100 9069 raft_consensus.cc:2468] T b85e2acbec2e485cb1b9728bd5763b0e P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:22.063202 9340 ts_tablet_manager.cc:1434] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.056s user 0.005s sys 0.001s
I20260926 01:56:22.073626 9341 tablet_bootstrap.cc:492] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:22.074137 9341 ts_tablet_manager.cc:1403] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.483s user 0.010s sys 0.001s
W20260926 01:56:22.058053 9003 leader_election.cc:343] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.064402 9339 tablet_bootstrap.cc:492] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:22.104463 10926 raft_consensus.cc:493] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.104643 9340 tablet_bootstrap.cc:492] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:22.104964 10926 raft_consensus.cc:515] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:22.107084 10926 leader_election.cc:290] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:22.083933 9006 leader_election.cc:343] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.107648 9006 leader_election.cc:304] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
W20260926 01:56:22.109010 9006 leader_election.cc:343] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.109414 9006 leader_election.cc:304] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.110615 10781 raft_consensus.cc:2749] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:22.110400 9341 raft_consensus.cc:359] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.111049 9341 raft_consensus.cc:385] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.111197 10781 raft_consensus.cc:493] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.111578 10781 raft_consensus.cc:515] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.111277 9341 raft_consensus.cc:740] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.113597 10781 leader_election.cc:290] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:22.114085 10781 raft_consensus.cc:2749] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:22.129763 9340 tablet_bootstrap.cc:654] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:22.121872 9339 tablet_bootstrap.cc:654] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:22.146718 9341 consensus_queue.cc:260] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.152176 9341 ts_tablet_manager.cc:1434] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
I20260926 01:56:22.154482 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5c66728bb54ec6beb4f0a2c25d9105" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:22.165136 9219 raft_consensus.cc:3060] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:22.175891 10913 raft_consensus.cc:493] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.176609 10913 raft_consensus.cc:515] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.179280 10913 leader_election.cc:290] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.180219 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:22.181613 8843 leader_election.cc:343] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:22.184911 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 85273.1us
I20260926 01:56:22.200102 9339 tablet_bootstrap.cc:492] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:22.200587 9339 ts_tablet_manager.cc:1403] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.136s user 0.008s sys 0.004s
I20260926 01:56:22.203328 9341 tablet_bootstrap.cc:492] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:22.204425 9340 tablet_bootstrap.cc:492] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:22.204900 9340 ts_tablet_manager.cc:1403] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.004s
I20260926 01:56:22.205590 9071 raft_consensus.cc:2468] T cdd0f4921da4436aa259486227a180d5 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:22.206995 8843 leader_election.cc:304] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.208734 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:22.209935 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:22.210625 9072 raft_consensus.cc:2468] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:22.225394 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5c66728bb54ec6beb4f0a2c25d9105" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:22.236822 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 112309us
I20260926 01:56:22.235538 9152 leader_election.cc:304] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:22.255132 9341 tablet_bootstrap.cc:654] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:22.257253 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:22.261539 9003 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 51644.4us
I20260926 01:56:22.270419 10942 raft_consensus.cc:2804] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:56:22.270620 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 63728.4us
I20260926 01:56:22.271032 10942 raft_consensus.cc:697] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:22.271400 8843 leader_election.cc:304] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.271867 10942 consensus_queue.cc:237] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.272451 8843 leader_election.cc:304] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.274344 10941 raft_consensus.cc:2804] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:22.274349 10914 raft_consensus.cc:2804] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:22.274796 10914 raft_consensus.cc:493] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:22.274850 10941 raft_consensus.cc:697] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:22.275142 10914 raft_consensus.cc:3060] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:22.275672 10941 consensus_queue.cc:237] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.285614 10914 raft_consensus.cc:515] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.287797 10914 leader_election.cc:290] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.289108 9339 raft_consensus.cc:359] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.289865 9339 raft_consensus.cc:385] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.290136 9339 raft_consensus.cc:740] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.291745 9340 raft_consensus.cc:359] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.292444 9340 raft_consensus.cc:385] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.292708 9340 raft_consensus.cc:740] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.298481 9340 consensus_queue.cc:260] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.303160 9339 consensus_queue.cc:260] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.304191 10968 raft_consensus.cc:493] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.304687 10968 raft_consensus.cc:515] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.307916 10968 leader_election.cc:290] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:22.307807 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
W20260926 01:56:22.309527 9003 leader_election.cc:343] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.299674 10851 raft_consensus.cc:2749] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:22.318027 9340 ts_tablet_manager.cc:1434] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.113s user 0.007s sys 0.000s
W20260926 01:56:22.319996 8843 leader_election.cc:343] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:22.322851 9006 leader_election.cc:343] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.323282 9006 leader_election.cc:304] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.324000 10968 raft_consensus.cc:2749] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:22.330120 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:22.331215 10941 raft_consensus.cc:493] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.331748 10941 raft_consensus.cc:515] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.333891 10941 leader_election.cc:290] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.335451 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:22.337291 8843 leader_election.cc:343] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.354916 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:22.355572 9069 raft_consensus.cc:3060] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:22.359576 8843 consensus_peers.cc:597] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:22.359740 9072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:22.360376 9072 raft_consensus.cc:2468] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:22.367015 8905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:22.372304 8843 leader_election.cc:304] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.373385 10941 raft_consensus.cc:2804] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:22.373728 10941 raft_consensus.cc:493] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:22.374035 10941 raft_consensus.cc:3060] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:22.378386 9069 raft_consensus.cc:2468] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
W20260926 01:56:22.380470 9003 leader_election.cc:343] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.385964 8843 leader_election.cc:304] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.386797 10914 raft_consensus.cc:2804] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:22.387235 10914 raft_consensus.cc:697] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:22.388019 10914 consensus_queue.cc:237] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.393208 10941 raft_consensus.cc:515] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.395952 9339 ts_tablet_manager.cc:1434] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.195s user 0.009s sys 0.000s
I20260926 01:56:22.399631 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:22.402617 10941 leader_election.cc:290] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.404704 9339 tablet_bootstrap.cc:492] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:22.405048 8843 leader_election.cc:343] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.417507 10941 consensus_queue.cc:1048] T 094b7485380b46fe8e1fef23ac50b830 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:22.403383 9006 leader_election.cc:343] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:22.443966 9006 leader_election.cc:304] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.400727 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:22.446409 9069 raft_consensus.cc:3060] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:22.447292 9340 tablet_bootstrap.cc:492] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:22.476521 10781 raft_consensus.cc:2749] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:22.492708 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 46790.3us
W20260926 01:56:22.516010 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 46464.7us
I20260926 01:56:22.535367 9340 tablet_bootstrap.cc:654] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:22.535713 9339 tablet_bootstrap.cc:654] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:22.543783 9341 tablet_bootstrap.cc:492] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:22.544319 9341 ts_tablet_manager.cc:1403] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.341s user 0.012s sys 0.007s
W20260926 01:56:22.545197 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.576s (SENT)
I20260926 01:56:22.545765 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
W20260926 01:56:22.564906 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 48330.1us
I20260926 01:56:22.567566 10985 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:22.587556 10987 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:22.588106 10987 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:22.620066 9341 raft_consensus.cc:359] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.620795 9341 raft_consensus.cc:385] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.621084 9341 raft_consensus.cc:740] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.621745 9341 consensus_queue.cc:260] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.651211 9069 raft_consensus.cc:2468] T 91964074a37441db8b09f78720159586 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:22.652658 8843 leader_election.cc:304] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:22.653528 10914 raft_consensus.cc:2804] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:22.654011 10914 raft_consensus.cc:697] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:22.654852 10914 consensus_queue.cc:237] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.687889 9339 tablet_bootstrap.cc:492] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:22.688494 9339 ts_tablet_manager.cc:1403] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.284s user 0.011s sys 0.000s
I20260926 01:56:22.691833 10987 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:22.694806 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:22.718176 10941 consensus_queue.cc:1048] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:22.727684 9340 tablet_bootstrap.cc:492] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:22.728250 9340 ts_tablet_manager.cc:1403] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.281s user 0.008s sys 0.003s
I20260926 01:56:22.728487 9339 raft_consensus.cc:359] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.729346 9339 raft_consensus.cc:385] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.729552 9339 raft_consensus.cc:740] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.730166 9339 consensus_queue.cc:260] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.732867 9341 ts_tablet_manager.cc:1434] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.188s user 0.008s sys 0.000s
W20260926 01:56:22.733199 8843 consensus_peers.cc:597] T 4efbc1e1ef35420db51742d99d5a32bf P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:22.734303 9341 tablet_bootstrap.cc:492] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:22.764328 10983 raft_consensus.cc:493] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.765110 10983 raft_consensus.cc:515] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.768091 10983 leader_election.cc:290] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.777992 9340 raft_consensus.cc:359] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.779139 9340 raft_consensus.cc:385] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:22.779505 9340 raft_consensus.cc:740] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:22.801021 9339 ts_tablet_manager.cc:1434] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.112s user 0.005s sys 0.000s
I20260926 01:56:22.802121 9339 tablet_bootstrap.cc:492] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:22.806602 10914 raft_consensus.cc:493] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.807155 10983 raft_consensus.cc:493] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:22.807191 10914 raft_consensus.cc:515] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.808037 10983 raft_consensus.cc:515] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:22.809379 10914 leader_election.cc:290] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.810312 10983 leader_election.cc:290] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:22.812256 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2156ef6b24c04ea38181832e7f18ff49" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:22.813884 9069 raft_consensus.cc:2468] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:22.815680 8843 leader_election.cc:304] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:22.816515 10914 raft_consensus.cc:2804] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:22.816867 10914 raft_consensus.cc:493] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:22.817195 10914 raft_consensus.cc:3060] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:22.780427 9340 consensus_queue.cc:260] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.858191 10968 raft_consensus.cc:493] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.858798 10968 raft_consensus.cc:515] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.861043 10968 leader_election.cc:290] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:22.810812 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2156ef6b24c04ea38181832e7f18ff49" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:22.864190 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:22.865018 9071 raft_consensus.cc:2393] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
W20260926 01:56:22.911918 9155 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:22.896319 10914 raft_consensus.cc:515] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.915092 10968 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.921986 8901 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:22.921576 10968 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.947248 8901 raft_consensus.cc:2393] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:22.950373 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:22.956102 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:22.965665 10996 raft_consensus.cc:493] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:22.966325 10996 raft_consensus.cc:515] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:22.971410 10996 leader_election.cc:290] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:22.973124 9339 tablet_bootstrap.cc:654] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:22.966333 10968 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:22.994606 9003 leader_election.cc:343] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.001576 10983 raft_consensus.cc:493] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:23.002101 10983 raft_consensus.cc:515] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.002655 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:23.004729 10983 leader_election.cc:290] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:23.005960 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:23.022207 10914 leader_election.cc:290] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:23.020401 9340 ts_tablet_manager.cc:1434] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.292s user 0.010s sys 0.000s
W20260926 01:56:23.019953 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 178080us
W20260926 01:56:23.045056 9006 leader_election.cc:343] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.045732 9006 leader_election.cc:304] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:23.046532 10968 raft_consensus.cc:2749] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:23.047851 9341 tablet_bootstrap.cc:654] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:23.052103 9152 leader_election.cc:304] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:23.055104 10851 raft_consensus.cc:3060] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:23.062503 10851 raft_consensus.cc:2749] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:23.066488 9152 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:23.075862 9340 tablet_bootstrap.cc:492] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:23.085484 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:23.091043 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:23.091789 9071 raft_consensus.cc:2468] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:23.094367 9339 tablet_bootstrap.cc:492] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:23.095000 9339 ts_tablet_manager.cc:1403] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.293s user 0.005s sys 0.007s
W20260926 01:56:23.097652 9006 leader_election.cc:343] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.099625 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2156ef6b24c04ea38181832e7f18ff49" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:23.101210 9339 raft_consensus.cc:359] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.102028 9339 raft_consensus.cc:385] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:23.102326 9339 raft_consensus.cc:740] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:23.103042 9339 consensus_queue.cc:260] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.106706 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:23.121677 10983 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:23.122375 10983 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:23.124578 10983 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:23.131548 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2156ef6b24c04ea38181832e7f18ff49" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:23.140647 9071 raft_consensus.cc:3060] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:23.147895 9071 raft_consensus.cc:2468] T 2156ef6b24c04ea38181832e7f18ff49 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:23.155048 9339 ts_tablet_manager.cc:1434] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.060s user 0.005s sys 0.001s
I20260926 01:56:23.156236 9339 tablet_bootstrap.cc:492] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:23.126267 9155 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
W20260926 01:56:23.170439 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 44210.1us
I20260926 01:56:23.173718 11006 raft_consensus.cc:493] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:23.174188 11006 raft_consensus.cc:515] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.176106 11006 leader_election.cc:290] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:23.177728 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6629d125861f4e27b61c10d585a35c10" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:23.184932 9003 leader_election.cc:343] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.185549 9003 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:23.186311 10992 raft_consensus.cc:2749] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:23.183492 8843 leader_election.cc:343] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.067349 9152 leader_election.cc:304] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
W20260926 01:56:23.193465 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 128093us
I20260926 01:56:23.194479 10851 raft_consensus.cc:2749] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:23.200043 9341 tablet_bootstrap.cc:492] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:23.200558 9341 ts_tablet_manager.cc:1403] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.467s user 0.018s sys 0.000s
I20260926 01:56:23.203294 9341 raft_consensus.cc:359] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.203948 9341 raft_consensus.cc:385] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:23.205363 9341 raft_consensus.cc:740] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:56:23.230698 8843 leader_election.cc:343] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:23.231490 8843 leader_election.cc:343] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:23.234839 8843 consensus_peers.cc:597] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:23.238632 9339 tablet_bootstrap.cc:654] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:23.258998 9218 raft_consensus.cc:3060] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:23.264381 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:23.265074 9220 raft_consensus.cc:2468] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:23.272630 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:23.273422 9069 raft_consensus.cc:2468] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
W20260926 01:56:23.310384 9152 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 99163.5us
I20260926 01:56:23.206023 9341 consensus_queue.cc:260] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.337081 9341 ts_tablet_manager.cc:1434] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.136s user 0.006s sys 0.000s
I20260926 01:56:23.339702 9341 tablet_bootstrap.cc:492] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:23.344359 9339 tablet_bootstrap.cc:492] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:23.346187 9339 ts_tablet_manager.cc:1403] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.190s user 0.011s sys 0.000s
I20260926 01:56:23.353431 9152 leader_election.cc:304] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:23.354215 10851 raft_consensus.cc:2749] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:23.359751 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_85a98e90b8d2451d9b35f8bf4013c05b [id=008c87e2e12f48519efdacc69466442a]
I20260926 01:56:23.361531 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f]
I20260926 01:56:23.365980 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]
W20260926 01:56:23.406931 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 48800.4us
I20260926 01:56:23.349582 9339 raft_consensus.cc:359] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.419093 9339 raft_consensus.cc:385] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:23.419526 9339 raft_consensus.cc:740] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:23.421684 9339 consensus_queue.cc:260] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.447471 9339 ts_tablet_manager.cc:1434] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.101s user 0.004s sys 0.002s
I20260926 01:56:23.463337 9339 tablet_bootstrap.cc:492] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:23.479660 11013 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:23.481869 11013 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:23.483044 10968 raft_consensus.cc:493] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:23.483619 10968 raft_consensus.cc:515] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.486028 10968 leader_election.cc:290] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:23.487982 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:23.489231 9006 leader_election.cc:343] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.491290 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:23.498608 11013 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:23.487713 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]
W20260926 01:56:23.509467 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 220706us
W20260926 01:56:23.513607 9152 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:23.515125 9152 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:23.519663 10851 raft_consensus.cc:2749] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:23.550926 9340 tablet_bootstrap.cc:654] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:23.562577 9339 tablet_bootstrap.cc:654] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:23.566776 9341 tablet_bootstrap.cc:654] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:23.581703 8843 leader_election.cc:304] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:23.603730 11006 raft_consensus.cc:2804] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260926 01:56:23.603974 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 74344.2us
I20260926 01:56:23.597299 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_730735d338704c849d0e9a76c846e6cc [id=0e58ec4f07b143c3a4d1b9b6bd5735da]
I20260926 01:56:23.604473 11006 raft_consensus.cc:493] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:23.606832 11006 raft_consensus.cc:3060] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:23.644049 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:23.660557 9341 tablet_bootstrap.cc:492] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:23.661048 9341 ts_tablet_manager.cc:1403] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.322s user 0.011s sys 0.005s
I20260926 01:56:23.668377 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]
I20260926 01:56:23.669935 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469]
I20260926 01:56:23.670755 9339 tablet_bootstrap.cc:492] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:23.671211 9339 ts_tablet_manager.cc:1403] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.208s user 0.010s sys 0.000s
I20260926 01:56:23.694028 10968 raft_consensus.cc:493] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:23.694555 10968 raft_consensus.cc:515] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.697072 10968 leader_election.cc:290] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:23.692200 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e]
I20260926 01:56:23.703547 9341 raft_consensus.cc:359] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.704329 9341 raft_consensus.cc:385] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:23.704608 9341 raft_consensus.cc:740] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:23.705291 9341 consensus_queue.cc:260] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.701252 9339 raft_consensus.cc:359] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.710042 9339 raft_consensus.cc:385] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:23.710357 9339 raft_consensus.cc:740] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:23.711032 9339 consensus_queue.cc:260] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.710501 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]
I20260926 01:56:23.714174 11006 raft_consensus.cc:515] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.716727 11006 leader_election.cc:290] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:23.716938 11014 consensus_queue.cc:1048] T 812fe778734d4210b744ed8e45d412f1 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:23.722000 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c]
I20260926 01:56:23.725699 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:23.739799 9003 leader_election.cc:343] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.740329 9003 leader_election.cc:304] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:23.741403 10781 raft_consensus.cc:2749] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:23.750200 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:23.743721 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]
W20260926 01:56:23.753549 9006 leader_election.cc:343] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:23.723277 8843 leader_election.cc:304] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:23.770797 11003 raft_consensus.cc:2804] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:23.771481 11003 raft_consensus.cc:697] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
W20260926 01:56:23.771528 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 53705.4us
I20260926 01:56:23.775022 11003 consensus_queue.cc:237] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.779670 11014 consensus_queue.cc:1048] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:23.792953 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0]
I20260926 01:56:23.801492 10996 raft_consensus.cc:493] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:23.802057 10996 raft_consensus.cc:515] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.804420 10996 leader_election.cc:290] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:23.822022 9340 tablet_bootstrap.cc:492] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:23.830420 9341 ts_tablet_manager.cc:1434] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.167s user 0.014s sys 0.001s
I20260926 01:56:23.846822 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_da5237bb06194562a023680e328f087d [id=1c21bf55cfd54a3ca9fcb01d02491bfe]
I20260926 01:56:23.877185 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bf92ccd2dbe1453d943f371ffec0badb [id=1cd337ec9ec94a47b24e774d0fff3321]
I20260926 01:56:23.887315 10982 raft_consensus.cc:493] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:23.887861 10982 raft_consensus.cc:515] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:23.890019 10982 leader_election.cc:290] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:23.891247 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:23.904987 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c]
I20260926 01:56:23.851197 9340 ts_tablet_manager.cc:1403] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.780s user 0.032s sys 0.015s
I20260926 01:56:23.935416 11003 consensus_queue.cc:1048] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:23.936041 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b]
I20260926 01:56:23.945726 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e85d17e9e48e4e54956e5c8460f7c9a8" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:23.880584 8843 leader_election.cc:304] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:23.924563 9341 tablet_bootstrap.cc:492] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:23.976183 11003 raft_consensus.cc:2804] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:56:23.976740 11003 raft_consensus.cc:493] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:23.977170 11003 raft_consensus.cc:3060] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:56:23.973876 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 93590.7us
I20260926 01:56:24.003353 9339 ts_tablet_manager.cc:1434] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.332s user 0.006s sys 0.001s
I20260926 01:56:24.003057 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa]
W20260926 01:56:24.026834 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 52404.9us
I20260926 01:56:24.029387 9152 leader_election.cc:304] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:24.030102 10983 raft_consensus.cc:2749] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:24.035249 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743]
W20260926 01:56:24.049422 9003 leader_election.cc:343] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:24.050048 9003 leader_election.cc:304] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:24.050801 10968 raft_consensus.cc:2749] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:24.075394 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.663s (SENT)
I20260926 01:56:24.075941 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:24.081249 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_0a43386150b14b53ad954d8f63768297 [id=27765f1075e54d509228a77011d6c8e8]
I20260926 01:56:24.082952 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e]
I20260926 01:56:24.084386 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181]
I20260926 01:56:24.091400 9340 raft_consensus.cc:359] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.092113 9340 raft_consensus.cc:385] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:24.092356 9340 raft_consensus.cc:740] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:24.092641 9341 tablet_bootstrap.cc:654] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:24.092989 9340 consensus_queue.cc:260] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.113852 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:24.114507 9216 raft_consensus.cc:3060] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:24.112841 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_57ad2cd7ce1e4bc6a071b8a0b985c3f2 [id=2af856781c904c77a11ca9f4966ca44b]
I20260926 01:56:24.145319 9340 ts_tablet_manager.cc:1434] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.230s user 0.010s sys 0.007s
I20260926 01:56:24.157574 9340 tablet_bootstrap.cc:492] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:24.159322 11020 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:24.161868 9216 raft_consensus.cc:2468] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
W20260926 01:56:24.164044 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 88702.6us
I20260926 01:56:24.204319 9341 tablet_bootstrap.cc:492] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:24.204857 9341 ts_tablet_manager.cc:1403] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.291s user 0.007s sys 0.009s
I20260926 01:56:24.207084 11003 raft_consensus.cc:515] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.209769 11003 leader_election.cc:290] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:24.204460 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f]
I20260926 01:56:24.213073 9340 tablet_bootstrap.cc:654] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:24.223192 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:24.263897 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1]
I20260926 01:56:24.286017 9219 raft_consensus.cc:3060] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:24.298388 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]
I20260926 01:56:24.300030 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb]
I20260926 01:56:24.242265 9341 raft_consensus.cc:359] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.305495 9341 raft_consensus.cc:385] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:24.305857 9341 raft_consensus.cc:740] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:24.306609 9341 consensus_queue.cc:260] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:24.328339 8843 consensus_peers.cc:597] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:24.329763 8843 consensus_peers.cc:597] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:24.336506 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 [id=30c67008e5de4b3da4aac5964c64c346]
I20260926 01:56:24.341851 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:24.342448 9069 raft_consensus.cc:3060] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:24.350158 9069 raft_consensus.cc:2468] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:24.353339 9067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:24.354007 9067 raft_consensus.cc:2468] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:24.308313 9341 ts_tablet_manager.cc:1434] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.103s user 0.004s sys 0.003s
I20260926 01:56:24.373317 9341 tablet_bootstrap.cc:492] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:24.395999 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e]
I20260926 01:56:24.400507 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6]
I20260926 01:56:24.402148 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4a4b59ee98dd4139b83d181413443621 [id=37400e3a991842198609512f0a9aeead]
W20260926 01:56:24.417627 8843 consensus_peers.cc:597] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:24.424619 9340 tablet_bootstrap.cc:492] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:24.425153 9340 ts_tablet_manager.cc:1403] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.268s user 0.010s sys 0.006s
I20260926 01:56:24.427886 9340 raft_consensus.cc:359] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.428620 9340 raft_consensus.cc:385] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:24.428900 9340 raft_consensus.cc:740] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:24.429584 9340 consensus_queue.cc:260] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:24.438777 8843 consensus_peers.cc:597] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:24.460268 8843 leader_election.cc:304] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:24.467993 10992 raft_consensus.cc:493] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:24.468492 10992 raft_consensus.cc:515] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.470655 10992 leader_election.cc:290] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:24.471107 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8]
I20260926 01:56:24.473147 9340 ts_tablet_manager.cc:1434] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.048s user 0.005s sys 0.007s
I20260926 01:56:24.473559 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff]
I20260926 01:56:24.480108 11024 raft_consensus.cc:2804] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:24.480672 11024 raft_consensus.cc:697] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:24.481539 11024 consensus_queue.cc:237] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.506559 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:24.499161 9340 tablet_bootstrap.cc:492] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:24.541798 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e]
W20260926 01:56:24.508075 8843 leader_election.cc:343] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:24.556252 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 49067.8us
I20260926 01:56:24.561141 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:24.561800 9220 raft_consensus.cc:3060] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:24.573544 9220 raft_consensus.cc:2468] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 2.
I20260926 01:56:24.581187 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410]
I20260926 01:56:24.587384 9341 tablet_bootstrap.cc:700] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532: Time spent opening tablet: real 0.209s user 0.004s sys 0.000s
I20260926 01:56:24.588284 9341 tablet_bootstrap.cc:654] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:24.591429 11019 raft_consensus.cc:493] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:24.591934 11019 raft_consensus.cc:515] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.594121 11019 leader_election.cc:290] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:24.596168 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f300124bc3a4177a606b116bfef3a65" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:24.619063 9340 tablet_bootstrap.cc:654] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:24.623667 11003 consensus_queue.cc:1048] T 91964074a37441db8b09f78720159586 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:24.624717 10942 consensus_queue.cc:1048] T cdd0f4921da4436aa259486227a180d5 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:24.625214 8843 leader_election.cc:304] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:24.626645 10996 raft_consensus.cc:2804] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:24.627007 10996 raft_consensus.cc:493] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:24.627292 10996 raft_consensus.cc:3060] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:24.628892 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe]
W20260926 01:56:24.634351 9006 leader_election.cc:343] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:24.648672 10996 raft_consensus.cc:515] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.651240 10996 leader_election.cc:290] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:24.649492 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932]
I20260926 01:56:24.668226 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f300124bc3a4177a606b116bfef3a65" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:24.672032 9152 leader_election.cc:304] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:24.672858 11019 raft_consensus.cc:2749] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:24.695976 9341 tablet_bootstrap.cc:492] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:24.696475 9341 ts_tablet_manager.cc:1403] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.323s user 0.010s sys 0.000s
I20260926 01:56:24.693337 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7]
I20260926 01:56:24.718541 9340 tablet_bootstrap.cc:492] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:24.719069 9340 ts_tablet_manager.cc:1403] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.228s user 0.015s sys 0.003s
W20260926 01:56:24.718860 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 58252.4us
I20260926 01:56:24.723165 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4]
I20260926 01:56:24.724624 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_818d89431f354c43ac77ba53fcb9cd8e [id=50846b080147404f91b349751918cca7]
I20260926 01:56:24.725993 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_05db3de2a0574423980de7d7b7098324 [id=5226bf119e6d4ffb839aea00e909e760]
W20260926 01:56:24.728782 8843 consensus_peers.cc:597] T b85e2acbec2e485cb1b9728bd5763b0e P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:24.732254 9220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:24.756706 9340 raft_consensus.cc:359] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.757517 9340 raft_consensus.cc:385] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:24.757778 9340 raft_consensus.cc:740] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:24.730154 8843 leader_election.cc:304] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532; no voters:
I20260926 01:56:24.765961 11003 raft_consensus.cc:2804] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:56:24.766566 11003 raft_consensus.cc:697] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [term 2 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:24.767505 11003 consensus_queue.cc:237] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.758445 9340 consensus_queue.cc:260] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.773699 9340 ts_tablet_manager.cc:1434] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20260926 01:56:24.782399 9219 raft_consensus.cc:3060] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:24.792914 9341 raft_consensus.cc:359] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.793715 9341 raft_consensus.cc:385] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:24.793987 9341 raft_consensus.cc:740] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:56:24.800278 8843 consensus_peers.cc:597] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:24.804648 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]
I20260926 01:56:24.844388 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a]
I20260926 01:56:24.794643 9341 consensus_queue.cc:260] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:24.858114 9341 ts_tablet_manager.cc:1434] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.161s user 0.007s sys 0.000s
I20260926 01:56:24.859150 9341 tablet_bootstrap.cc:492] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:24.867532 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 49472.6us
I20260926 01:56:24.951375 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:24.987541 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.738s (SENT)
I20260926 01:56:24.988008 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:25.019294 9340 tablet_bootstrap.cc:492] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
W20260926 01:56:25.027812 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 147906us
I20260926 01:56:25.032919 11044 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:25.057869 9341 tablet_bootstrap.cc:654] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:25.067447 8843 leader_election.cc:343] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:25.095520 9340 tablet_bootstrap.cc:654] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:25.122588 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0f4e81776f942ff96a9c1f274996c6a" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:25.123416 9071 raft_consensus.cc:3060] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:25.131089 9071 raft_consensus.cc:2468] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 2.
I20260926 01:56:25.142518 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]
I20260926 01:56:25.206007 11019 raft_consensus.cc:493] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:25.206550 11019 raft_consensus.cc:515] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:25.208490 11019 leader_election.cc:290] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:25.222226 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d]
I20260926 01:56:25.223099 9210 raft_consensus.cc:3060] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:25.223968 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc]
I20260926 01:56:25.226215 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393]
I20260926 01:56:25.212477 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:25.237135 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7]
I20260926 01:56:25.239845 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353]
I20260926 01:56:25.242774 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]
I20260926 01:56:25.244601 11048 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:25.245713 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e04a4278262e4d7f8d7a94ddf8666bbe [id=7879fa789a5143a6a1bb95768cf1bf50]
I20260926 01:56:25.256229 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd]
I20260926 01:56:25.259136 11035 raft_consensus.cc:493] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:25.259624 11035 raft_consensus.cc:515] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:25.261623 11035 leader_election.cc:290] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:25.262059 10996 raft_consensus.cc:493] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:25.265667 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264]
I20260926 01:56:25.284229 11048 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:25.288569 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:25.289160 9068 raft_consensus.cc:3060] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:25.263569 11043 raft_consensus.cc:493] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:25.297013 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:25.305073 11048 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:25.338974 9068 raft_consensus.cc:2468] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:25.293752 11043 raft_consensus.cc:515] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:25.263717 10996 raft_consensus.cc:515] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:25.355918 11043 leader_election.cc:290] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:25.357263 10996 leader_election.cc:290] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:25.386005 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]
I20260926 01:56:25.388847 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2]
W20260926 01:56:25.395753 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 132311us
I20260926 01:56:25.414999 9341 tablet_bootstrap.cc:492] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:25.415666 9341 ts_tablet_manager.cc:1403] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.557s user 0.010s sys 0.002s
I20260926 01:56:25.418491 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "094b7485380b46fe8e1fef23ac50b830" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:25.420894 9340 tablet_bootstrap.cc:492] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:25.421403 9340 ts_tablet_manager.cc:1403] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.402s user 0.014s sys 0.019s
I20260926 01:56:25.390344 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588]
I20260926 01:56:25.462359 9152 leader_election.cc:304] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:25.463358 11019 raft_consensus.cc:2749] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:25.504379 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]
I20260926 01:56:25.473317 9340 raft_consensus.cc:359] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:25.525506 9340 raft_consensus.cc:385] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:25.508939 9341 raft_consensus.cc:359] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:25.532533 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b]
W20260926 01:56:25.549201 9155 consensus_peers.cc:597] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcb000f9234d4382ad019f63015b92e7. This is attempt 6: this message will repeat every 5th retry.
W20260926 01:56:25.551977 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 94193us
W20260926 01:56:25.563395 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 98803.3us
I20260926 01:56:25.579643 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654]
I20260926 01:56:25.528369 9340 raft_consensus.cc:740] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:25.537631 9341 raft_consensus.cc:385] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:25.608479 9341 raft_consensus.cc:740] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:25.608825 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6]
I20260926 01:56:25.618479 9215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:25.640246 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f]
I20260926 01:56:25.603391 9340 consensus_queue.cc:260] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:25.674186 9003 leader_election.cc:343] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:25.674381 9340 ts_tablet_manager.cc:1434] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.253s user 0.013s sys 0.001s
I20260926 01:56:25.674813 9003 leader_election.cc:304] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:25.676743 11036 raft_consensus.cc:2749] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:25.612265 9341 consensus_queue.cc:260] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:25.710233 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:25.720332 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:25.753399 9341 ts_tablet_manager.cc:1434] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.337s user 0.011s sys 0.003s
I20260926 01:56:25.758682 9341 tablet_bootstrap.cc:492] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:25.881945 9340 tablet_bootstrap.cc:492] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:25.892716 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:25.893393 9071 raft_consensus.cc:2468] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:25.937525 9340 tablet_bootstrap.cc:654] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:25.952440 9910 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.665s (SENT) [suppressed 1 similar messages]
I20260926 01:56:25.967216 9061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:25.971295 9061 raft_consensus.cc:2468] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:25.995704 10982 raft_consensus.cc:493] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:25.996261 10982 raft_consensus.cc:515] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.000854 9061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:25.998373 10982 leader_election.cc:290] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:26.027302 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:26.028118 9073 raft_consensus.cc:2468] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:26.003985 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdd0f4921da4436aa259486227a180d5" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:26.081382 9152 leader_election.cc:304] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:26.082659 10982 raft_consensus.cc:2749] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:26.067641 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 61253.2us
I20260926 01:56:26.087697 11014 consensus_queue.cc:1048] T c450d101aa2b47c0bac3d4964c7bd6bd P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:26.089767 11014 consensus_queue.cc:1048] T 2156ef6b24c04ea38181832e7f18ff49 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:26.135578 10891 raft_consensus.cc:493] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:26.136077 10891 raft_consensus.cc:515] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.138055 10891 leader_election.cc:290] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:26.146049 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:26.157226 9006 leader_election.cc:343] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.098133 8843 leader_election.cc:304] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
W20260926 01:56:26.171348 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 73299.9us
I20260926 01:56:26.178678 11014 raft_consensus.cc:2804] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:26.179360 11014 raft_consensus.cc:697] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:26.180294 11014 consensus_queue.cc:237] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.182802 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]
I20260926 01:56:26.185150 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e]
I20260926 01:56:26.186364 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83]
I20260926 01:56:26.189126 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29]
I20260926 01:56:26.190383 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b]
I20260926 01:56:26.191672 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a]
I20260926 01:56:26.193756 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d]
I20260926 01:56:26.233845 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]
I20260926 01:56:26.251489 9340 tablet_bootstrap.cc:492] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:26.253122 9340 ts_tablet_manager.cc:1403] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.372s user 0.005s sys 0.009s
W20260926 01:56:26.262923 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 56356.9us
I20260926 01:56:26.265123 10982 raft_consensus.cc:493] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:26.265676 10982 raft_consensus.cc:515] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.271481 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:26.277725 10982 leader_election.cc:290] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:26.278849 10982 raft_consensus.cc:493] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:26.283592 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b4ea0a3f8a40eb90fd545ca6df7a8e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:26.281476 10982 raft_consensus.cc:515] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:26.290911 10982 leader_election.cc:290] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:26.292778 8899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:26.323885 9340 raft_consensus.cc:359] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.340873 9341 tablet_bootstrap.cc:654] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:26.359304 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e76071c66b1d4635ac1a28110eb684fa" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:26.376756 9155 leader_election.cc:304] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:26.378185 9155 leader_election.cc:304] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:26.381517 9340 raft_consensus.cc:385] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:26.381907 9340 raft_consensus.cc:740] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:26.382850 9340 consensus_queue.cc:260] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:26.308578 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 48457.3us
I20260926 01:56:26.363371 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:26.392493 11071 raft_consensus.cc:2749] T 00b4ea0a3f8a40eb90fd545ca6df7a8e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:26.393659 11071 raft_consensus.cc:2749] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:26.416144 11036 raft_consensus.cc:493] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:26.416630 11036 raft_consensus.cc:515] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.420109 11036 leader_election.cc:290] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:26.427210 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e]
I20260926 01:56:26.432989 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]
I20260926 01:56:26.437963 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]
I20260926 01:56:26.448416 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:26.454018 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 69287.8us
I20260926 01:56:26.472656 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]
I20260926 01:56:26.474143 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9]
I20260926 01:56:26.487201 9340 ts_tablet_manager.cc:1434] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.234s user 0.005s sys 0.001s
I20260926 01:56:26.484119 11014 raft_consensus.cc:493] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:26.490489 11014 raft_consensus.cc:515] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.492466 11014 leader_election.cc:290] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:26.493047 8843 leader_election.cc:304] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:26.493920 11003 raft_consensus.cc:2804] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:26.494248 11003 raft_consensus.cc:493] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:26.494586 11003 raft_consensus.cc:3060] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:26.496338 8843 leader_election.cc:304] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:26.497336 11065 raft_consensus.cc:2804] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:26.497632 11065 raft_consensus.cc:493] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:26.497927 11065 raft_consensus.cc:3060] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:26.506891 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:26.512600 8843 leader_election.cc:304] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:26.513531 11024 raft_consensus.cc:2804] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:26.513913 11024 raft_consensus.cc:493] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:26.514313 11024 raft_consensus.cc:3060] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:26.523887 9155 consensus_peers.cc:597] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16f49a5a10ea4f7a821d89f18e3af560. This is attempt 11: this message will repeat every 5th retry.
I20260926 01:56:26.534029 11060 raft_consensus.cc:493] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:26.534739 11060 raft_consensus.cc:515] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.536931 11060 leader_election.cc:290] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:26.540077 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 88954.4us
I20260926 01:56:26.557542 11003 raft_consensus.cc:515] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.559887 11003 leader_election.cc:290] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:26.571555 11065 raft_consensus.cc:515] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.573662 11065 leader_election.cc:290] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:26.579309 9006 leader_election.cc:343] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.587410 9340 tablet_bootstrap.cc:492] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:26.592300 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:26.604712 11080 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:26.605352 11080 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
W20260926 01:56:26.623601 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 44323.5us
W20260926 01:56:26.576472 9003 leader_election.cc:343] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.626190 9003 leader_election.cc:304] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:26.643193 11024 raft_consensus.cc:515] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.645531 11024 leader_election.cc:290] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:26.647471 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639]
W20260926 01:56:26.651315 9006 leader_election.cc:343] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:26.655905 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.665s (SENT)
I20260926 01:56:26.656379 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:26.673046 11080 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:26.674810 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:26.685560 11082 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260926 01:56:26.689100 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 112646us
I20260926 01:56:26.690665 10891 raft_consensus.cc:2749] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:26.697328 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204]
I20260926 01:56:26.701179 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7d32da88077d4060913cc8ae86df83f6 [id=d2357d68ebb749358cb06a52c588b99f]
I20260926 01:56:26.703544 9003 leader_election.cc:304] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:26.708333 9210 raft_consensus.cc:3060] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:26.718566 11060 raft_consensus.cc:2749] T c450d101aa2b47c0bac3d4964c7bd6bd P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:26.749938 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:26.755378 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:26.645787 9340 tablet_bootstrap.cc:654] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:26.791268 11079 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:26.791779 11079 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.792964 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c [id=d680dfae1d21415382be575ea6696a44]
I20260926 01:56:26.794061 11079 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:26.797447 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:26.802095 9006 leader_election.cc:343] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.804695 9073 raft_consensus.cc:1275] T d0f4e81776f942ff96a9c1f274996c6a P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Refusing update from remote peer 49bd320bf73f4252804ec867ec077510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260926 01:56:26.728453 9341 tablet_bootstrap.cc:492] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:26.838500 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96]
I20260926 01:56:26.841346 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff]
W20260926 01:56:26.867475 8843 leader_election.cc:343] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:26.868846 8843 leader_election.cc:343] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.869113 11057 consensus_queue.cc:1048] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:26.874748 8843 leader_election.cc:343] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.848583 9341 ts_tablet_manager.cc:1403] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 1.090s user 0.054s sys 0.029s
I20260926 01:56:26.889974 9340 tablet_bootstrap.cc:492] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:26.890498 9340 ts_tablet_manager.cc:1403] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.303s user 0.004s sys 0.012s
I20260926 01:56:26.897441 9340 raft_consensus.cc:359] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:26.903973 9340 raft_consensus.cc:385] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:26.904253 9340 raft_consensus.cc:740] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:26.904884 9340 consensus_queue.cc:260] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:26.909117 8843 leader_election.cc:343] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.911530 9340 ts_tablet_manager.cc:1434] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260926 01:56:26.914829 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]
I20260926 01:56:26.915264 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:26.917497 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:26.921408 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:26.923220 9340 tablet_bootstrap.cc:492] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510: Bootstrap starting.
I20260926 01:56:26.926889 9210 raft_consensus.cc:1275] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Refusing update from remote peer 49bd320bf73f4252804ec867ec077510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260926 01:56:26.947727 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146]
W20260926 01:56:26.965864 8843 leader_election.cc:343] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:26.973052 8843 leader_election.cc:343] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:26.974828 8843 leader_election.cc:343] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:26.976675 11065 consensus_queue.cc:1048] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:26.978245 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894]
I20260926 01:56:26.972404 9340 tablet_bootstrap.cc:654] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:26.979652 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b]
I20260926 01:56:26.980945 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90]
I20260926 01:56:27.021943 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c]
I20260926 01:56:27.075415 9340 tablet_bootstrap.cc:492] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510: No bootstrap required, opened a new log
I20260926 01:56:27.077044 9340 ts_tablet_manager.cc:1403] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510: Time spent bootstrapping tablet: real 0.154s user 0.008s sys 0.004s
I20260926 01:56:27.082032 9340 raft_consensus.cc:359] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.093230 9340 raft_consensus.cc:385] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:27.094573 9340 raft_consensus.cc:740] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Initialized, Role: FOLLOWER
I20260926 01:56:27.095434 9340 consensus_queue.cc:260] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.087404 9341 raft_consensus.cc:359] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.110214 9340 ts_tablet_manager.cc:1434] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260926 01:56:27.111862 9341 raft_consensus.cc:385] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:27.117043 9341 raft_consensus.cc:740] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:27.127058 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c]
I20260926 01:56:27.128464 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]
I20260926 01:56:27.131099 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]
I20260926 01:56:27.130343 9341 consensus_queue.cc:260] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.158424 11024 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:27.158993 11024 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.161841 11024 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:27.164232 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
W20260926 01:56:27.165731 8843 leader_election.cc:343] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:27.205789 9061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:27.206911 9061 raft_consensus.cc:3060] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:27.227145 9341 ts_tablet_manager.cc:1434] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.312s user 0.023s sys 0.010s
I20260926 01:56:27.236392 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1]
W20260926 01:56:27.248802 8599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.512s (SENT) [suppressed 1 similar messages]
I20260926 01:56:27.275471 9059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:27.275683 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:27.276505 9059 raft_consensus.cc:3060] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:27.285863 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:27.286926 9069 raft_consensus.cc:2468] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:27.276521 9068 raft_consensus.cc:3060] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:56:27.298492 8843 consensus_peers.cc:597] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:27.304366 11101 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
W20260926 01:56:27.309962 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 61348.4us
I20260926 01:56:27.298297 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad]
I20260926 01:56:27.332907 9341 tablet_bootstrap.cc:492] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:27.333477 11101 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:27.341060 9341 tablet_bootstrap.cc:654] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:27.367311 9059 raft_consensus.cc:2468] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:27.367208 9065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:27.368227 9065 raft_consensus.cc:2468] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:27.378777 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:27.393785 11101 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:27.394598 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]
I20260926 01:56:27.397043 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d]
I20260926 01:56:27.378130 9068 raft_consensus.cc:2468] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:27.438664 9061 raft_consensus.cc:2468] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:27.451480 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]
W20260926 01:56:27.509078 9155 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 52452.7us
I20260926 01:56:27.514433 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4]
W20260926 01:56:27.514673 9003 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.76.188:38143, user_credentials={real_user=slave}} blocked reactor thread for 50535.7us
W20260926 01:56:27.533516 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 82783.6us
I20260926 01:56:27.551323 11024 raft_consensus.cc:493] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:27.570003 11024 raft_consensus.cc:515] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.577883 11057 raft_consensus.cc:493] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:27.578377 11057 raft_consensus.cc:515] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.580669 11057 leader_election.cc:290] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:27.573431 11024 leader_election.cc:290] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:27.596487 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:27.598178 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:27.615653 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:27.616482 8903 raft_consensus.cc:2468] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 0.
W20260926 01:56:27.629369 9155 consensus_peers.cc:597] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e12827ad93c47cf9013e238215ceb7a. This is attempt 11: this message will repeat every 5th retry.
I20260926 01:56:27.654704 11003 consensus_queue.cc:1048] T d0f4e81776f942ff96a9c1f274996c6a P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:27.656114 8843 leader_election.cc:343] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:27.656800 8843 leader_election.cc:343] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:27.719293 11024 consensus_queue.cc:1048] T dd22a450f5914523871a088b24795e9f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:27.833132 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.512s (SENT)
I20260926 01:56:27.833690 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:27.856730 11116 raft_consensus.cc:493] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:27.857264 11116 raft_consensus.cc:515] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:27.848977 9341 tablet_bootstrap.cc:492] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:27.863559 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:27.865296 11116 leader_election.cc:290] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:27.849375 8843 leader_election.cc:304] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
W20260926 01:56:27.878244 9003 leader_election.cc:343] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:27.879657 9003 leader_election.cc:304] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:27.868263 9341 ts_tablet_manager.cc:1403] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.535s user 0.031s sys 0.060s
I20260926 01:56:27.878235 11024 raft_consensus.cc:2804] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:27.898922 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4efbc1e1ef35420db51742d99d5a32bf" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:27.901654 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 68592.2us
I20260926 01:56:27.901999 11121 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:27.903110 9152 leader_election.cc:304] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:27.904580 11116 raft_consensus.cc:2749] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:27.900539 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 53733.9us
I20260926 01:56:27.927453 11096 raft_consensus.cc:2749] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:56:27.928542 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 45716.1us
I20260926 01:56:27.893344 11024 raft_consensus.cc:493] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260926 01:56:27.942528 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 64342us
I20260926 01:56:27.947284 11024 raft_consensus.cc:3060] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:27.983068 9003 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:27.983973 11111 raft_consensus.cc:2804] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:27.984303 11111 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:27.984587 11111 raft_consensus.cc:3060] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:27.994735 11111 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.027401 9210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:28.033290 10398 tablet_loader.cc:96] loaded metadata for tablet 030f4d683e404fbd8d2e6863e18f71c0 (table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72])
W20260926 01:56:28.037997 9006 leader_election.cc:343] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:28.071259 10398 tablet_loader.cc:96] loaded metadata for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_57ad2cd7ce1e4bc6a071b8a0b985c3f2 [id=2af856781c904c77a11ca9f4966ca44b])
I20260926 01:56:28.089802 11111 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:28.096918 11024 raft_consensus.cc:515] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.094581 10398 tablet_loader.cc:96] loaded metadata for tablet 05042bcc0a6e426d9e3e8b25d8d55540 (table default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469])
I20260926 01:56:28.106137 10398 tablet_loader.cc:96] loaded metadata for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f])
I20260926 01:56:28.120391 10398 tablet_loader.cc:96] loaded metadata for tablet 08b1d98089fd499a8e1ee217b561a801 (table default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0])
I20260926 01:56:28.127558 10398 tablet_loader.cc:96] loaded metadata for tablet 094b7485380b46fe8e1fef23ac50b830 (table default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181])
I20260926 01:56:28.077700 9341 raft_consensus.cc:359] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.132017 9341 raft_consensus.cc:385] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:28.138628 9341 raft_consensus.cc:740] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:28.174023 10398 tablet_loader.cc:96] loaded metadata for tablet 0ae1f842516445588886ed7e7149c02a (table default.Table_7d32da88077d4060913cc8ae86df83f6 [id=d2357d68ebb749358cb06a52c588b99f])
I20260926 01:56:28.175437 10398 tablet_loader.cc:96] loaded metadata for tablet 0b2a99b3bfd64925a82236cec1d9e71f (table default.Table_05db3de2a0574423980de7d7b7098324 [id=5226bf119e6d4ffb839aea00e909e760])
I20260926 01:56:28.142891 8843 leader_election.cc:304] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:28.183084 11057 raft_consensus.cc:2804] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:56:28.183175 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 43580.7us
I20260926 01:56:28.183624 11057 raft_consensus.cc:697] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:28.184142 8843 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:28.184501 11057 consensus_queue.cc:237] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.185284 11122 raft_consensus.cc:2804] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:28.185616 11122 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:28.185917 11122 raft_consensus.cc:3060] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:28.187419 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:28.198973 11024 leader_election.cc:290] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:28.209112 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:28.209793 9071 raft_consensus.cc:2468] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:28.211195 9071 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:28.211766 9071 raft_consensus.cc:2468] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 0.
I20260926 01:56:28.202281 11122 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.165938 9341 consensus_queue.cc:260] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.214053 8903 raft_consensus.cc:2393] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d20f26c9295641ecbf77d290ca206bb2 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:28.224303 8843 leader_election.cc:304] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:28.225311 11003 raft_consensus.cc:2804] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:28.227499 11024 consensus_queue.cc:1048] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260926 01:56:28.245424 10398 tablet_loader.cc:96] loaded metadata for tablet 0fdb7a7328c54031bd3afac1636d0f75 (table default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e])
I20260926 01:56:28.247644 11122 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:28.279057 11003 raft_consensus.cc:697] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
W20260926 01:56:28.279522 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 53681us
I20260926 01:56:28.280391 11003 consensus_queue.cc:237] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.299299 9341 ts_tablet_manager.cc:1434] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.409s user 0.011s sys 0.011s
I20260926 01:56:28.303025 10398 tablet_loader.cc:96] loaded metadata for tablet 1279a644e6604fb0bc20f53cd23dd8be (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a])
I20260926 01:56:28.304247 10398 tablet_loader.cc:96] loaded metadata for tablet 14926a101b404f6b8d5a35b2aadc6c39 (table default.Table_bf92ccd2dbe1453d943f371ffec0badb [id=1cd337ec9ec94a47b24e774d0fff3321])
I20260926 01:56:28.368149 10398 tablet_loader.cc:96] loaded metadata for tablet 1b84d9bc9c6f41bd80482c129e569648 (table default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b])
I20260926 01:56:28.371933 9341 tablet_bootstrap.cc:492] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:28.376724 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 91336.1us
I20260926 01:56:28.377878 9341 tablet_bootstrap.cc:654] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:28.393704 8843 leader_election.cc:304] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:28.395632 11065 raft_consensus.cc:2804] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:28.396183 11065 raft_consensus.cc:697] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:28.397047 11065 consensus_queue.cc:237] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.405326 10398 tablet_loader.cc:96] loaded metadata for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c (table default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a])
I20260926 01:56:28.406668 10398 tablet_loader.cc:96] loaded metadata for tablet 1c5c66728bb54ec6beb4f0a2c25d9105 (table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c])
I20260926 01:56:28.412515 9341 tablet_bootstrap.cc:492] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:28.412990 9341 ts_tablet_manager.cc:1403] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.006s
I20260926 01:56:28.415722 9341 raft_consensus.cc:359] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.416404 9341 raft_consensus.cc:385] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:28.416651 9341 raft_consensus.cc:740] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:28.417249 9341 consensus_queue.cc:260] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.431881 9341 ts_tablet_manager.cc:1434] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260926 01:56:28.433036 9341 tablet_bootstrap.cc:492] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:28.437549 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 60144.5us
I20260926 01:56:28.439829 9341 tablet_bootstrap.cc:654] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:28.447515 10398 tablet_loader.cc:96] loaded metadata for tablet 1c8761a6c3f7435193a91bfc4fe7fa86 (table default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8])
I20260926 01:56:28.455819 10398 tablet_loader.cc:96] loaded metadata for tablet 1f10cb68586b4a73abb86641b90c12f7 (table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f])
I20260926 01:56:28.467870 9341 tablet_bootstrap.cc:492] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:28.468355 9341 ts_tablet_manager.cc:1403] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.004s
W20260926 01:56:28.468353 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 69578.9us
I20260926 01:56:28.469061 8843 leader_election.cc:304] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:28.470360 8843 leader_election.cc:304] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:28.470491 11003 raft_consensus.cc:2804] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:28.470846 11003 raft_consensus.cc:493] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:28.471146 11003 raft_consensus.cc:3060] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:28.476507 9003 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:28.480788 9341 raft_consensus.cc:359] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.481470 9341 raft_consensus.cc:385] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:28.481707 9341 raft_consensus.cc:740] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:28.482311 9341 consensus_queue.cc:260] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.486814 9341 ts_tablet_manager.cc:1434] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260926 01:56:28.487742 9341 tablet_bootstrap.cc:492] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:28.479408 11003 raft_consensus.cc:515] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.497802 11003 leader_election.cc:290] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:28.500211 10398 tablet_loader.cc:96] loaded metadata for tablet 215a061440854c56b00df9fda9192598 (table default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e])
I20260926 01:56:28.507006 10398 tablet_loader.cc:96] loaded metadata for tablet 225bcc7bd4884188bba09b331b5da8e2 (table default.Table_0a43386150b14b53ad954d8f63768297 [id=27765f1075e54d509228a77011d6c8e8])
I20260926 01:56:28.512290 10398 tablet_loader.cc:96] loaded metadata for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a])
I20260926 01:56:28.525183 11133 raft_consensus.cc:2749] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 01:56:28.527895 10398 tablet_loader.cc:96] loaded metadata for tablet 272bb87fe22c4720bcb3f0e07482dd8b (table default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1])
W20260926 01:56:28.530849 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 54423.3us
I20260926 01:56:28.534019 10398 tablet_loader.cc:96] loaded metadata for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409])
I20260926 01:56:28.544070 11129 raft_consensus.cc:2804] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:56:28.544565 11129 raft_consensus.cc:493] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:28.544925 11129 raft_consensus.cc:3060] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:28.546275 10398 tablet_loader.cc:96] loaded metadata for tablet 2a093fc10b604aa491b6e0f9596a589d (table default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e])
I20260926 01:56:28.555830 11129 raft_consensus.cc:515] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.557868 11129 leader_election.cc:290] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:28.558734 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 87351.1us
I20260926 01:56:28.556226 10398 tablet_loader.cc:96] loaded metadata for tablet 2c0cd3e30f084966b27921e658cacf58 (table default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146])
W20260926 01:56:28.572225 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 144395us
I20260926 01:56:28.579092 10398 tablet_loader.cc:96] loaded metadata for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd])
I20260926 01:56:28.583280 10398 tablet_loader.cc:96] loaded metadata for tablet 38a52e650df5449c86924387a0535d39 (table default.Table_e04a4278262e4d7f8d7a94ddf8666bbe [id=7879fa789a5143a6a1bb95768cf1bf50])
I20260926 01:56:28.585302 10398 tablet_loader.cc:96] loaded metadata for tablet 3a7f7eaf2d444886894b3b39902be4c9 (table default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d])
I20260926 01:56:28.588318 10398 tablet_loader.cc:96] loaded metadata for tablet 3c26f557ac274973824e6f9a4532de74 (table default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b])
I20260926 01:56:28.589653 10398 tablet_loader.cc:96] loaded metadata for tablet 3c7151ffe5d94a0599bbe3e6c8999371 (table default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c])
I20260926 01:56:28.590988 10398 tablet_loader.cc:96] loaded metadata for tablet 3f1d71caf78f4c75b144af14dce19d70 (table default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4])
I20260926 01:56:28.592460 10398 tablet_loader.cc:96] loaded metadata for tablet 4411120140f3410c9f4e33e26da20d98 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d])
I20260926 01:56:28.594612 10398 tablet_loader.cc:96] loaded metadata for tablet 4623456312284a698ae69d4054909647 (table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa])
I20260926 01:56:28.597704 9341 tablet_bootstrap.cc:654] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:28.604283 10398 tablet_loader.cc:96] loaded metadata for tablet 4efbc1e1ef35420db51742d99d5a32bf (table default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410])
I20260926 01:56:28.605700 10398 tablet_loader.cc:96] loaded metadata for tablet 504cca4587d64bc4964b298ba1a74fe9 (table default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393])
I20260926 01:56:28.607846 10398 tablet_loader.cc:96] loaded metadata for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b])
W20260926 01:56:28.607334 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 47709.9us
I20260926 01:56:28.609550 10398 tablet_loader.cc:96] loaded metadata for tablet 56b0e40e5e964b90bd2dd158bbc4497a (table default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad])
I20260926 01:56:28.612613 10398 tablet_loader.cc:96] loaded metadata for tablet 572b97ad3aef4dce8cb3bcb521e90e48 (table default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 [id=30c67008e5de4b3da4aac5964c64c346])
I20260926 01:56:28.614777 10398 tablet_loader.cc:96] loaded metadata for tablet 5cbf646569e6497dac90e308f0681372 (table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b])
I20260926 01:56:28.617525 10398 tablet_loader.cc:96] loaded metadata for tablet 5da0c41e2e19409d938f8f8ec29cb91a (table default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c])
I20260926 01:56:28.618883 10398 tablet_loader.cc:96] loaded metadata for tablet 5eef6a48e87e4c61a23058942a82bccc (table default.Table_4a4b59ee98dd4139b83d181413443621 [id=37400e3a991842198609512f0a9aeead])
I20260926 01:56:28.620532 10398 tablet_loader.cc:96] loaded metadata for tablet 5f300124bc3a4177a606b116bfef3a65 (table default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff])
I20260926 01:56:28.622640 10398 tablet_loader.cc:96] loaded metadata for tablet 661148692f9e4127af9abf5cc6cc0f54 (table default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1])
I20260926 01:56:28.624011 10398 tablet_loader.cc:96] loaded metadata for tablet 6629d125861f4e27b61c10d585a35c10 (table default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588])
I20260926 01:56:28.626909 10398 tablet_loader.cc:96] loaded metadata for tablet 6752df6a58914df5b8718dd0439aa885 (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63])
I20260926 01:56:28.628113 10398 tablet_loader.cc:96] loaded metadata for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2])
I20260926 01:56:28.629268 10398 tablet_loader.cc:96] loaded metadata for tablet 6ab082379974486bbea9f87f5aa64e10 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da])
I20260926 01:56:28.636404 10398 tablet_loader.cc:96] loaded metadata for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b (table default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b])
I20260926 01:56:28.639016 10398 tablet_loader.cc:96] loaded metadata for tablet 6f7419abcc2c482e97df70f03e047450 (table default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d])
I20260926 01:56:28.641809 10398 tablet_loader.cc:96] loaded metadata for tablet 70d2447335b34731a4cb2cd0f3b4bf13 (table default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e])
I20260926 01:56:28.646275 10398 tablet_loader.cc:96] loaded metadata for tablet 7e57300d2bcf4e0496ed13a4e7d967f3 (table default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2])
I20260926 01:56:28.647516 11140 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:28.648414 11140 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:28.653151 10398 tablet_loader.cc:96] loaded metadata for tablet 812fe778734d4210b744ed8e45d412f1 (table default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe])
I20260926 01:56:28.654583 9210 raft_consensus.cc:3060] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:28.656628 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:28.687206 9341 tablet_bootstrap.cc:492] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:28.687716 9341 ts_tablet_manager.cc:1403] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.200s user 0.007s sys 0.008s
I20260926 01:56:28.695287 9341 raft_consensus.cc:359] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.696031 9341 raft_consensus.cc:385] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:28.696281 9341 raft_consensus.cc:740] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:28.696940 9341 consensus_queue.cc:260] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.698429 9341 ts_tablet_manager.cc:1434] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260926 01:56:28.699417 9341 tablet_bootstrap.cc:492] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:28.705003 9219 raft_consensus.cc:3060] T 2156ef6b24c04ea38181832e7f18ff49 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:28.705314 9341 tablet_bootstrap.cc:654] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:28.729250 11140 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:28.731231 10398 tablet_loader.cc:96] loaded metadata for tablet 8711788a85f94c599224123f6b8cc814 (table default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c])
W20260926 01:56:28.731838 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.682s (SENT)
I20260926 01:56:28.732312 9341 tablet_bootstrap.cc:492] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:28.732326 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:28.732765 9341 ts_tablet_manager.cc:1403] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.006s
I20260926 01:56:28.735353 9341 raft_consensus.cc:359] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.736025 9341 raft_consensus.cc:385] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:28.736269 9341 raft_consensus.cc:740] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:28.736933 9341 consensus_queue.cc:260] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.743512 10398 tablet_loader.cc:96] loaded metadata for tablet 880d77d42c394e6a9cf0b3768a915402 (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4])
I20260926 01:56:28.745061 10398 tablet_loader.cc:96] loaded metadata for tablet 89fb965257de4db6b043b4a6dbe22709 (table default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc])
I20260926 01:56:28.750572 9341 ts_tablet_manager.cc:1434] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260926 01:56:28.751694 9341 tablet_bootstrap.cc:492] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:28.757584 9341 tablet_bootstrap.cc:654] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:28.758445 11136 raft_consensus.cc:493] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:28.759096 11136 raft_consensus.cc:515] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.762646 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:28.772714 9341 tablet_bootstrap.cc:492] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:28.773208 9341 ts_tablet_manager.cc:1403] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.001s
I20260926 01:56:28.774230 11136 leader_election.cc:290] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:28.775717 11147 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:28.775935 9341 raft_consensus.cc:359] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.776571 9341 raft_consensus.cc:385] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 01:56:28.776657 9910 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 44865.7us
I20260926 01:56:28.776803 9341 raft_consensus.cc:740] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:28.777390 9341 consensus_queue.cc:260] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.784997 9341 ts_tablet_manager.cc:1434] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260926 01:56:28.785980 9341 tablet_bootstrap.cc:492] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:28.791818 9341 tablet_bootstrap.cc:654] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:28.803210 10398 tablet_loader.cc:96] loaded metadata for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791])
I20260926 01:56:28.807762 10398 tablet_loader.cc:96] loaded metadata for tablet 8aae2a23d3fe478e928757db4b30f5ca (table default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e])
I20260926 01:56:28.814244 10398 tablet_loader.cc:96] loaded metadata for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204])
I20260926 01:56:28.831691 10398 tablet_loader.cc:96] loaded metadata for tablet 91964074a37441db8b09f78720159586 (table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9])
I20260926 01:56:28.848625 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b85e2acbec2e485cb1b9728bd5763b0e" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:28.867272 11111 raft_consensus.cc:493] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:28.868085 11111 raft_consensus.cc:515] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.871088 11111 leader_election.cc:290] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:28.874867 9210 raft_consensus.cc:3060] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:28.887674 9216 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:28.898049 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:28.902369 9211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:28.907210 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:28.915572 9215 raft_consensus.cc:3060] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:28.924113 10398 tablet_loader.cc:96] loaded metadata for tablet 9d973538ff534f05b301cb619475d4f3 (table default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a])
I20260926 01:56:28.924376 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:28.945521 9341 tablet_bootstrap.cc:492] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:28.946029 9341 ts_tablet_manager.cc:1403] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.160s user 0.016s sys 0.004s
I20260926 01:56:28.948559 9341 raft_consensus.cc:359] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.949203 9341 raft_consensus.cc:385] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:28.949458 9341 raft_consensus.cc:740] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:28.950074 9341 consensus_queue.cc:260] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:28.952159 9341 ts_tablet_manager.cc:1434] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 01:56:28.953012 9341 tablet_bootstrap.cc:492] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:28.926270 10398 tablet_loader.cc:96] loaded metadata for tablet 9db85241141f4a54a2b7dce15412a6f9 (table default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff])
I20260926 01:56:28.963773 10398 tablet_loader.cc:96] loaded metadata for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_85a98e90b8d2451d9b35f8bf4013c05b [id=008c87e2e12f48519efdacc69466442a])
I20260926 01:56:28.972030 9341 tablet_bootstrap.cc:654] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:28.966259 10398 tablet_loader.cc:96] loaded metadata for tablet a6e8059b94564c42894c6c8a3d695950 (table default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c [id=d680dfae1d21415382be575ea6696a44])
I20260926 01:56:28.992358 10398 tablet_loader.cc:96] loaded metadata for tablet a73602570425443d97d28fcebe3cc6fc (table default.Table_818d89431f354c43ac77ba53fcb9cd8e [id=50846b080147404f91b349751918cca7])
I20260926 01:56:29.038417 9069 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:29.039079 9069 raft_consensus.cc:3060] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:29.047456 9069 raft_consensus.cc:2468] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:29.049216 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:29.049938 9068 raft_consensus.cc:2393] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 49bd320bf73f4252804ec867ec077510 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:29.060547 9063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:29.061136 9063 raft_consensus.cc:3060] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:29.069319 9063 raft_consensus.cc:2468] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:29.073992 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "49bd320bf73f4252804ec867ec077510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:29.075707 9068 raft_consensus.cc:3060] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:29.117946 10398 tablet_loader.cc:96] loaded metadata for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576])
I20260926 01:56:29.117812 9213 raft_consensus.cc:3060] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:29.127296 9218 raft_consensus.cc:3060] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:29.140930 9341 tablet_bootstrap.cc:492] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:29.141597 9341 ts_tablet_manager.cc:1403] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.189s user 0.021s sys 0.013s
I20260926 01:56:29.147357 9068 raft_consensus.cc:2468] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bd320bf73f4252804ec867ec077510 in term 1.
I20260926 01:56:29.149962 10398 tablet_loader.cc:96] loaded metadata for tablet accc62f46ce24a4d97743dca05e65f51 (table default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7])
I20260926 01:56:29.172431 10398 tablet_loader.cc:96] loaded metadata for tablet af0840d6761a4521b0b6136b80eb6a6c (table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f])
I20260926 01:56:29.173887 10398 tablet_loader.cc:96] loaded metadata for tablet b25107dd5c104c83a2a21847bb2fc24f (table default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894])
W20260926 01:56:29.200328 8843 leader_election.cc:343] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:29.201152 8843 leader_election.cc:343] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:29.165766 9341 raft_consensus.cc:359] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.201984 9341 raft_consensus.cc:385] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 01:56:29.222913 8843 leader_election.cc:343] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:56:29.224875 8843 leader_election.cc:343] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:56:29.211272 9341 raft_consensus.cc:740] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:29.241724 9341 consensus_queue.cc:260] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.243593 9341 ts_tablet_manager.cc:1434] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.102s user 0.006s sys 0.003s
I20260926 01:56:29.244827 9341 tablet_bootstrap.cc:492] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:29.251495 10398 tablet_loader.cc:96] loaded metadata for tablet b778c95327fc477ca210cb3c983665c1 (table default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1])
I20260926 01:56:29.274307 10398 tablet_loader.cc:96] loaded metadata for tablet b85e2acbec2e485cb1b9728bd5763b0e (table default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264])
W20260926 01:56:29.276895 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 51387.3us
W20260926 01:56:29.277070 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 193304us
I20260926 01:56:29.289991 10398 tablet_loader.cc:96] loaded metadata for tablet c448caeeedfb4fc88eee88e40eebfeeb (table default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353])
I20260926 01:56:29.291385 10398 tablet_loader.cc:96] loaded metadata for tablet c450d101aa2b47c0bac3d4964c7bd6bd (table default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639])
I20260926 01:56:29.294170 10398 tablet_loader.cc:96] loaded metadata for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94])
I20260926 01:56:29.278827 9152 leader_election.cc:304] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:29.327474 10398 tablet_loader.cc:96] loaded metadata for tablet cdd0f4921da4436aa259486227a180d5 (table default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29])
I20260926 01:56:29.330231 10398 tablet_loader.cc:96] loaded metadata for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1])
I20260926 01:56:29.341536 10398 tablet_loader.cc:96] loaded metadata for tablet d0f4e81776f942ff96a9c1f274996c6a (table default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96])
I20260926 01:56:29.344911 10398 tablet_loader.cc:96] loaded metadata for tablet d7e2b22ca59745879270e8dc20f2a4fa (table default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d])
I20260926 01:56:29.362949 11136 raft_consensus.cc:493] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:29.363435 9341 tablet_bootstrap.cc:654] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:29.364693 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 86213.7us
W20260926 01:56:29.366930 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 71060.8us
I20260926 01:56:29.371178 10398 tablet_loader.cc:96] loaded metadata for tablet dbf222f9e14d4332b679f7ee94af1e0e (table default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6])
I20260926 01:56:29.374706 10398 tablet_loader.cc:96] loaded metadata for tablet dd22a450f5914523871a088b24795e9f (table default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90])
I20260926 01:56:29.378240 10398 tablet_loader.cc:96] loaded metadata for tablet e140cf11b8bb49f9a94c0158a2afca4d (table default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83])
I20260926 01:56:29.387303 10398 tablet_loader.cc:96] loaded metadata for tablet e18ddaa6f0e748cf930e29cbca3844b1 (table default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b])
I20260926 01:56:29.363534 11136 raft_consensus.cc:515] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.399572 11136 leader_election.cc:290] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:29.404383 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 95969.8us
I20260926 01:56:29.404958 9059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:29.408255 11137 raft_consensus.cc:493] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:29.408891 11137 raft_consensus.cc:515] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.430640 10398 tablet_loader.cc:96] loaded metadata for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9])
I20260926 01:56:29.435964 9208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd22a450f5914523871a088b24795e9f" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:29.436451 10398 tablet_loader.cc:96] loaded metadata for tablet e55a31bb9b9d4862b6bcef71df0cdaea (table default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932])
I20260926 01:56:29.451957 11157 raft_consensus.cc:2749] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:29.452503 10398 tablet_loader.cc:96] loaded metadata for tablet e723112c1a6a4a6c89b6d17491d67c65 (table default.Table_da5237bb06194562a023680e328f087d [id=1c21bf55cfd54a3ca9fcb01d02491bfe])
I20260926 01:56:29.453394 11137 leader_election.cc:290] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:29.454964 10398 tablet_loader.cc:96] loaded metadata for tablet e7512e1d08e74ef6a2c699352347eb8d (table default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654])
I20260926 01:56:29.456579 10398 tablet_loader.cc:96] loaded metadata for tablet e76071c66b1d4635ac1a28110eb684fa (table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb])
I20260926 01:56:29.458798 10398 tablet_loader.cc:96] loaded metadata for tablet e85d17e9e48e4e54956e5c8460f7c9a8 (table default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e])
I20260926 01:56:29.459851 11134 raft_consensus.cc:493] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:29.460301 10398 tablet_loader.cc:96] loaded metadata for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7])
I20260926 01:56:29.460361 11134 raft_consensus.cc:515] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.461542 10398 tablet_loader.cc:96] loaded metadata for tablet e8f5c3bdad494f34a3b72ab693599c55 (table default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc])
I20260926 01:56:29.462813 10398 tablet_loader.cc:96] loaded metadata for tablet ea75f309937e4ee49b35558117847b05 (table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4])
I20260926 01:56:29.464311 10398 tablet_loader.cc:96] loaded metadata for tablet ec749c375d784786ba93020996708bb1 (table default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6])
I20260926 01:56:29.465396 10398 tablet_loader.cc:96] loaded metadata for tablet edb7afa012944b3ba3c34bd994a95a83 (table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134])
I20260926 01:56:29.466655 10398 tablet_loader.cc:96] loaded metadata for tablet ee418347ebb04927913810d539ef57ea (table default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743])
I20260926 01:56:29.468750 10398 tablet_loader.cc:96] loaded metadata for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f])
I20260926 01:56:29.470343 10398 tablet_loader.cc:96] loaded metadata for tablet f2ee16b0c3e84936abba8248f194786a (table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c])
I20260926 01:56:29.472427 10398 tablet_loader.cc:96] loaded metadata for tablet f633f937b6c747f78efc6d3e167e83f0 (table default.Table_730735d338704c849d0e9a76c846e6cc [id=0e58ec4f07b143c3a4d1b9b6bd5735da])
I20260926 01:56:29.472429 9216 raft_consensus.cc:3060] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:29.477998 10398 tablet_loader.cc:96] loaded metadata for tablet f74618f7c46744d2a08cb7b4e389c9f3 (table default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4])
I20260926 01:56:29.479291 10398 tablet_loader.cc:96] loaded metadata for tablet f7dcb2358e434f36a831352105863807 (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f])
I20260926 01:56:29.496357 11134 leader_election.cc:290] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:29.508255 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:29.510000 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
W20260926 01:56:29.533041 10398 catalog_manager.cc:1525] Time spent T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loading table and tablet metadata into memory: real 21.444s user 0.515s sys 0.173s
I20260926 01:56:29.533516 10398 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:56:29.534730 8549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:29.535866 8549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 5) took 21150 ms (client timeout 516 ms). Trace:
W20260926 01:56:29.536257 8549 rpcz_store.cc:269] 0926 01:56:08.385263 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:08.385427 (+ 164us) service_pool.cc:224] Handling call
0926 01:56:29.535757 (+21150330us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260926 01:56:29.537485 8549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 11) took 15739 ms (client timeout 569 ms). Trace:
W20260926 01:56:29.537786 8549 rpcz_store.cc:269] 0926 01:56:13.797798 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.537247 (+15739449us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.537418 (+ 171us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.538745 8546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
W20260926 01:56:29.539578 8546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 8) took 20444 ms (client timeout 568 ms). Trace:
W20260926 01:56:29.539883 8546 rpcz_store.cc:269] 0926 01:56:09.095008 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:09.095239 (+ 231us) service_pool.cc:224] Handling call
0926 01:56:29.539512 (+20444273us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260926 01:56:29.540759 8546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 12) took 15040 ms (client timeout 569 ms). Trace:
W20260926 01:56:29.541074 8546 rpcz_store.cc:269] 0926 01:56:14.500221 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.540567 (+15040346us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.540692 (+ 125us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.541877 8546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 33) took 14627 ms (client timeout 594 ms). Trace:
I20260926 01:56:29.543295 8550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:29.544335 8550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 28) took 21109 ms (client timeout 704 ms). Trace:
W20260926 01:56:29.544691 8550 rpcz_store.cc:269] 0926 01:56:08.434605 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:08.435346 (+ 741us) service_pool.cc:224] Handling call
0926 01:56:29.544245 (+21108899us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260926 01:56:29.545552 8550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 13) took 13694 ms (client timeout 589 ms). Trace:
W20260926 01:56:29.545843 8550 rpcz_store.cc:269] 0926 01:56:15.850879 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.545390 (+13694511us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.545500 (+ 110us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.547106 11134 raft_consensus.cc:493] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:29.542733 8547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:29.547565 11134 raft_consensus.cc:515] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:29.548049 8547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 7) took 20593 ms (client timeout 516 ms). Trace:
W20260926 01:56:29.548339 8547 rpcz_store.cc:269] 0926 01:56:08.954606 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:08.955250 (+ 644us) service_pool.cc:224] Handling call
0926 01:56:29.547980 (+20592730us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:29.548641 9341 tablet_bootstrap.cc:492] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:29.549081 9341 ts_tablet_manager.cc:1403] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.305s user 0.008s sys 0.014s
W20260926 01:56:29.549149 8547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 34) took 12967 ms (client timeout 507 ms). Trace:
W20260926 01:56:29.549414 8547 rpcz_store.cc:269] 0926 01:56:16.581992 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.548992 (+12967000us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.549094 (+ 102us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.549574 11134 leader_election.cc:290] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:29.550192 8547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 35) took 12439 ms (client timeout 507 ms). Trace:
I20260926 01:56:29.550721 8545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:29.551596 8545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 29) took 20275 ms (client timeout 704 ms). Trace:
W20260926 01:56:29.551890 8545 rpcz_store.cc:269] 0926 01:56:09.276489 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:09.276784 (+ 295us) service_pool.cc:224] Handling call
0926 01:56:29.551527 (+20274743us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260926 01:56:29.552683 8545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 36) took 10794 ms (client timeout 617 ms). Trace:
W20260926 01:56:29.552979 8545 rpcz_store.cc:269] 0926 01:56:18.758226 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.552536 (+10794310us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.552639 (+ 103us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.553750 8545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 15) took 10346 ms (client timeout 600 ms). Trace:
W20260926 01:56:29.554030 8545 rpcz_store.cc:269] 0926 01:56:19.207531 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.553592 (+10346061us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.553706 (+ 114us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.556988 9341 raft_consensus.cc:359] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.557646 9341 raft_consensus.cc:385] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:29.557888 9341 raft_consensus.cc:740] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
W20260926 01:56:29.558533 8547 rpcz_store.cc:269] 0926 01:56:17.111313 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.550039 (+12438726us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.550143 (+ 104us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.542176 8546 rpcz_store.cc:269] 0926 01:56:14.914978 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.541711 (+14626733us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.541830 (+ 119us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.560554 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 91656.5us
I20260926 01:56:29.563228 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:29.563807 9068 raft_consensus.cc:2468] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 1.
I20260926 01:56:29.565814 8843 leader_election.cc:304] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:29.566553 11152 raft_consensus.cc:2804] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:29.566993 11152 raft_consensus.cc:697] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:29.567770 11152 consensus_queue.cc:237] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.578611 9155 leader_election.cc:304] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
W20260926 01:56:29.578908 8549 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 32) took 15291 ms (client timeout 594 ms). Trace:
W20260926 01:56:29.579246 8549 rpcz_store.cc:269] 0926 01:56:14.284175 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.559191 (+15275016us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.575217 (+ 16026us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.564347 8541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:29.581059 8550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 14) took 13130 ms (client timeout 589 ms). Trace:
W20260926 01:56:29.581383 8550 rpcz_store.cc:269] 0926 01:56:16.444904 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.558486 (+13113582us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.575094 (+ 16608us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.583071 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:29.585181 8544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 9) took 17803 ms (client timeout 509 ms). Trace:
W20260926 01:56:29.585511 8544 rpcz_store.cc:269] 0926 01:56:11.781532 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:11.781733 (+ 201us) service_pool.cc:224] Handling call
0926 01:56:29.585105 (+17803372us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:29.585706 11136 raft_consensus.cc:2804] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:56:29.585744 8843 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:29.586043 11136 raft_consensus.cc:493] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:29.586400 11136 raft_consensus.cc:3060] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:56:29.586413 8544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 39) took 7618 ms (client timeout 576 ms). Trace:
W20260926 01:56:29.586714 8544 rpcz_store.cc:269] 0926 01:56:21.967888 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.586238 (+7618350us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.586348 (+ 110us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.587451 8544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 17) took 6895 ms (client timeout 663 ms). Trace:
W20260926 01:56:29.587721 8544 rpcz_store.cc:269] 0926 01:56:22.692308 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.587285 (+6894977us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.587391 (+ 106us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.588500 8544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 18) took 6163 ms (client timeout 663 ms). Trace:
W20260926 01:56:29.588765 8544 rpcz_store.cc:269] 0926 01:56:23.425503 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.588357 (+6162854us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.588455 (+ 98us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.589226 11122 raft_consensus.cc:2749] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260926 01:56:29.589504 8544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 40) took 6103 ms (client timeout 738 ms). Trace:
W20260926 01:56:29.589764 8544 rpcz_store.cc:269] 0926 01:56:23.485744 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.589356 (+6103612us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.589459 (+ 103us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.590605 8548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 8 candidate_status { last_received { term: 7 index: 357 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
W20260926 01:56:29.591473 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 6) took 21067 ms (client timeout 568 ms). Trace:
W20260926 01:56:29.591763 8548 rpcz_store.cc:269] 0926 01:56:08.524093 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:08.524313 (+ 220us) service_pool.cc:224] Handling call
0926 01:56:29.591404 (+21067091us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:29.590492 9341 consensus_queue.cc:260] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:29.592548 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 19) took 4305 ms (client timeout 665 ms). Trace:
W20260926 01:56:29.592828 8548 rpcz_store.cc:269] 0926 01:56:25.287530 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.592411 (+4304881us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.592500 (+ 89us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.593607 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 20) took 3603 ms (client timeout 665 ms). Trace:
W20260926 01:56:29.593894 8548 rpcz_store.cc:269] 0926 01:56:25.989762 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.593462 (+3603700us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.593546 (+ 84us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.594623 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 42) took 2920 ms (client timeout 512 ms). Trace:
W20260926 01:56:29.594914 8548 rpcz_store.cc:269] 0926 01:56:26.673987 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.594495 (+2920508us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.594579 (+ 84us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.595566 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 43) took 2275 ms (client timeout 512 ms). Trace:
W20260926 01:56:29.595855 8548 rpcz_store.cc:269] 0926 01:56:27.320420 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.595440 (+2275020us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.595521 (+ 81us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.596503 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 21) took 2260 ms (client timeout 682 ms). Trace:
W20260926 01:56:29.596779 8548 rpcz_store.cc:269] 0926 01:56:27.336450 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.596379 (+2259929us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.596461 (+ 82us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.597496 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 22) took 1548 ms (client timeout 682 ms). Trace:
W20260926 01:56:29.597783 8548 rpcz_store.cc:269] 0926 01:56:28.049498 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.597370 (+1547872us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.597453 (+ 83us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.598207 8843 leader_election.cc:304] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:29.598803 8539 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:42048 (request call id 332) took 21480 ms. Trace:
I20260926 01:56:29.599160 11122 raft_consensus.cc:2804] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:56:29.599609 8545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 37) took 10154 ms (client timeout 617 ms). Trace:
I20260926 01:56:29.599627 11122 raft_consensus.cc:697] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
W20260926 01:56:29.599917 8545 rpcz_store.cc:269] 0926 01:56:19.429734 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.582850 (+10153116us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.583927 (+ 1077us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.599112 8539 rpcz_store.cc:276] 0926 01:56:08.118618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:08.122356 (+ 3738us) service_pool.cc:224] Handling call
0926 01:56:29.598572 (+21476216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:29.582226 8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:29.601812 8546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 16) took 9616 ms (client timeout 600 ms). Trace:
W20260926 01:56:29.602131 8546 rpcz_store.cc:269] 0926 01:56:19.968374 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.583486 (+9615112us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.584935 (+ 1449us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.600493 11122 consensus_queue.cc:237] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
W20260926 01:56:29.602957 8543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 30) took 17474 ms (client timeout 612 ms). Trace:
W20260926 01:56:29.603260 8543 rpcz_store.cc:269] 0926 01:56:12.128466 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:12.129015 (+ 549us) service_pool.cc:224] Handling call
0926 01:56:29.602883 (+17473868us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:29.580245 8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:29.604854 8542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 10) took 17280 ms (client timeout 509 ms). Trace:
W20260926 01:56:29.605155 8542 rpcz_store.cc:269] 0926 01:56:12.324109 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:12.324323 (+ 214us) service_pool.cc:224] Handling call
0926 01:56:29.604771 (+17280448us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260926 01:56:29.606236 8541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 31) took 16799 ms (client timeout 612 ms). Trace:
W20260926 01:56:29.606567 8541 rpcz_store.cc:269] 0926 01:56:12.807152 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:12.807403 (+ 251us) service_pool.cc:224] Handling call
0926 01:56:29.606161 (+16798758us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260926 01:56:29.607754 8550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 38) took 8571 ms (client timeout 576 ms). Trace:
W20260926 01:56:29.608063 8550 rpcz_store.cc:269] 0926 01:56:21.035915 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.605595 (+8569680us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.607696 (+ 2101us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.608846 9341 ts_tablet_manager.cc:1434] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.059s user 0.007s sys 0.001s
I20260926 01:56:29.609953 9341 tablet_bootstrap.cc:492] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:29.613260 8545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:29.618506 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
W20260926 01:56:29.625340 8544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 41) took 5341 ms (client timeout 738 ms). Trace:
W20260926 01:56:29.625749 8544 rpcz_store.cc:269] 0926 01:56:24.249298 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.590338 (+5341040us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.590426 (+ 88us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260926 01:56:29.631884 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 44) took 980 ms (client timeout 643 ms). Trace:
W20260926 01:56:29.632215 8548 rpcz_store.cc:269] 0926 01:56:28.651796 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.598368 (+946572us) service_pool.cc:211] Skipping call since client already timed out
0926 01:56:29.631799 (+ 33431us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260926 01:56:29.631603 10398 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:56:29.634621 10398 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
W20260926 01:56:29.640952 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 65925.1us
I20260926 01:56:29.605942 11136 raft_consensus.cc:515] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.642545 11174 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260926 01:56:29.645129 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2"
I20260926 01:56:29.645750 9068 raft_consensus.cc:3060] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:29.652722 11134 raft_consensus.cc:493] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:29.653285 11134 raft_consensus.cc:515] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.656024 10398 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:56:29.656255 11134 leader_election.cc:290] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:29.656492 9059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:29.657366 9059 raft_consensus.cc:2393] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
W20260926 01:56:29.668870 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 44165.2us
I20260926 01:56:29.670852 8843 leader_election.cc:304] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters: bc00feec3f564608851f0112e97ab532
I20260926 01:56:29.674163 11057 raft_consensus.cc:2804] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:56:29.674576 11136 leader_election.cc:290] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 election: Requested vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:29.675302 10398 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loaded TSK: 0
I20260926 01:56:29.676362 10398 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:56:29.676537 9068 raft_consensus.cc:2468] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc00feec3f564608851f0112e97ab532 in term 2.
I20260926 01:56:29.679607 9341 tablet_bootstrap.cc:654] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:29.684100 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35ffd8b0a9a44ae18a2d0e383a5b7724" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:29.686995 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91964074a37441db8b09f78720159586" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:29.692032 9155 leader_election.cc:304] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:29.696380 9006 leader_election.cc:304] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d20f26c9295641ecbf77d290ca206bb2; no voters: 49bd320bf73f4252804ec867ec077510, bc00feec3f564608851f0112e97ab532
I20260926 01:56:29.700357 11169 raft_consensus.cc:2749] T dd22a450f5914523871a088b24795e9f P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:29.702437 11135 raft_consensus.cc:2804] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:56:29.702996 11135 raft_consensus.cc:697] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 2 LEADER]: Becoming Leader. State: Replica: bc00feec3f564608851f0112e97ab532, State: Running, Role: LEADER
I20260926 01:56:29.703832 11135 consensus_queue.cc:237] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.707288 10398 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:56:29.736651 9341 tablet_bootstrap.cc:492] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:29.754933 9341 ts_tablet_manager.cc:1403] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.145s user 0.012s sys 0.000s
I20260926 01:56:29.772711 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_85a98e90b8d2451d9b35f8bf4013c05b [id=008c87e2e12f48519efdacc69466442a]
I20260926 01:56:29.774832 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f]
I20260926 01:56:29.776644 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]
I20260926 01:56:29.779371 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]
I20260926 01:56:29.782124 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_730735d338704c849d0e9a76c846e6cc [id=0e58ec4f07b143c3a4d1b9b6bd5735da]
I20260926 01:56:29.783875 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]
I20260926 01:56:29.787851 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469]
I20260926 01:56:29.789733 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e]
I20260926 01:56:29.791677 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]
I20260926 01:56:29.792227 9341 raft_consensus.cc:359] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.792917 9341 raft_consensus.cc:385] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:29.793179 9341 raft_consensus.cc:740] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:29.793562 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c]
I20260926 01:56:29.793810 9341 consensus_queue.cc:260] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.795363 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]
I20260926 01:56:29.798106 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0]
I20260926 01:56:29.799990 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_da5237bb06194562a023680e328f087d [id=1c21bf55cfd54a3ca9fcb01d02491bfe]
I20260926 01:56:29.801679 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bf92ccd2dbe1453d943f371ffec0badb [id=1cd337ec9ec94a47b24e774d0fff3321]
I20260926 01:56:29.802363 11057 raft_consensus.cc:697] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Leader. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:29.803529 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c]
I20260926 01:56:29.805387 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b]
I20260926 01:56:29.807301 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa]
I20260926 01:56:29.809152 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743]
I20260926 01:56:29.810918 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_0a43386150b14b53ad954d8f63768297 [id=27765f1075e54d509228a77011d6c8e8]
I20260926 01:56:29.812892 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e]
I20260926 01:56:29.814756 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181]
I20260926 01:56:29.815356 9341 ts_tablet_manager.cc:1434] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20260926 01:56:29.816591 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_57ad2cd7ce1e4bc6a071b8a0b985c3f2 [id=2af856781c904c77a11ca9f4966ca44b]
I20260926 01:56:29.819480 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f]
I20260926 01:56:29.820667 9341 tablet_bootstrap.cc:492] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:29.822351 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1]
I20260926 01:56:29.804064 11057 consensus_queue.cc:237] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.825325 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]
I20260926 01:56:29.827198 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb]
I20260926 01:56:29.828949 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 [id=30c67008e5de4b3da4aac5964c64c346]
I20260926 01:56:29.831887 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e]
W20260926 01:56:29.832643 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 125169us
I20260926 01:56:29.833760 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6]
I20260926 01:56:29.835575 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4a4b59ee98dd4139b83d181413443621 [id=37400e3a991842198609512f0a9aeead]
I20260926 01:56:29.836115 11177 consensus_queue.cc:1048] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:29.838548 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8]
I20260926 01:56:29.841459 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff]
I20260926 01:56:29.844362 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e]
I20260926 01:56:29.847309 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410]
I20260926 01:56:29.851639 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe]
I20260926 01:56:29.853559 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932]
W20260926 01:56:29.855077 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 54561.3us
I20260926 01:56:29.855355 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7]
I20260926 01:56:29.857093 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4]
I20260926 01:56:29.858956 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_818d89431f354c43ac77ba53fcb9cd8e [id=50846b080147404f91b349751918cca7]
I20260926 01:56:29.861043 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_05db3de2a0574423980de7d7b7098324 [id=5226bf119e6d4ffb839aea00e909e760]
I20260926 01:56:29.862874 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]
I20260926 01:56:29.864591 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a]
I20260926 01:56:29.870700 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]
I20260926 01:56:29.862004 9152 leader_election.cc:304] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:29.872419 9152 leader_election.cc:304] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:29.872625 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d]
I20260926 01:56:29.874375 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc]
I20260926 01:56:29.877210 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393]
I20260926 01:56:29.878938 9341 tablet_bootstrap.cc:654] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:29.880159 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7]
I20260926 01:56:29.881973 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353]
I20260926 01:56:29.887090 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]
I20260926 01:56:29.891247 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e04a4278262e4d7f8d7a94ddf8666bbe [id=7879fa789a5143a6a1bb95768cf1bf50]
I20260926 01:56:29.893242 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd]
I20260926 01:56:29.902233 11177 consensus_queue.cc:1048] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:29.905077 11186 raft_consensus.cc:2749] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:29.905627 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264]
I20260926 01:56:29.893129 11135 raft_consensus.cc:2749] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:29.911486 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]
I20260926 01:56:29.915062 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2]
I20260926 01:56:29.926489 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588]
I20260926 01:56:29.931176 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]
I20260926 01:56:29.933040 9341 tablet_bootstrap.cc:492] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:29.933530 9341 ts_tablet_manager.cc:1403] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.117s user 0.008s sys 0.004s
I20260926 01:56:29.934988 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b]
I20260926 01:56:29.938006 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654]
I20260926 01:56:29.939857 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6]
I20260926 01:56:29.941665 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f]
I20260926 01:56:29.939939 9341 raft_consensus.cc:359] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.942623 9341 raft_consensus.cc:385] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:29.944665 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]
I20260926 01:56:29.949463 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e]
I20260926 01:56:29.949967 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c450d101aa2b47c0bac3d4964c7bd6bd" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:29.951366 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83]
I20260926 01:56:29.951548 9341 raft_consensus.cc:740] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:29.952354 9341 consensus_queue.cc:260] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.953969 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29]
I20260926 01:56:29.956881 11134 raft_consensus.cc:493] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:29.957315 11134 raft_consensus.cc:515] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:29.961591 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:29.966225 11134 leader_election.cc:290] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:29.971746 8843 consensus_peers.cc:597] T d7e2b22ca59745879270e8dc20f2a4fa P 49bd320bf73f4252804ec867ec077510 -> Peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Couldn't send request to peer bc00feec3f564608851f0112e97ab532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:29.975368 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b]
I20260926 01:56:29.982357 9341 ts_tablet_manager.cc:1434] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260926 01:56:29.984527 11184 consensus_queue.cc:1048] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:29.984086 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a]
I20260926 01:56:29.991560 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:29.992380 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70d2447335b34731a4cb2cd0f3b4bf13" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:29.997228 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea75f309937e4ee49b35558117847b05" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:29.997362 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d]
I20260926 01:56:30.004853 8904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c1d70856f5c4ac68a5fcc8b5e1ef89c" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:30.005008 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]
I20260926 01:56:30.009203 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e]
I20260926 01:56:30.011104 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]
I20260926 01:56:29.996187 9341 tablet_bootstrap.cc:492] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:30.017333 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]
I20260926 01:56:30.019405 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]
I20260926 01:56:30.021339 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9]
I20260926 01:56:30.023165 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639]
I20260926 01:56:30.025066 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204]
I20260926 01:56:30.028484 9152 leader_election.cc:304] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:30.030359 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7d32da88077d4060913cc8ae86df83f6 [id=d2357d68ebb749358cb06a52c588b99f]
I20260926 01:56:30.030721 9152 leader_election.cc:304] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:30.033569 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c [id=d680dfae1d21415382be575ea6696a44]
I20260926 01:56:30.035575 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96]
I20260926 01:56:30.038373 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff]
I20260926 01:56:30.040289 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]
I20260926 01:56:30.040584 9215 raft_consensus.cc:3060] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:30.042135 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146]
I20260926 01:56:30.043503 9211 raft_consensus.cc:3060] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:30.043753 11116 raft_consensus.cc:2749] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:30.044051 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894]
I20260926 01:56:30.045394 11157 raft_consensus.cc:3060] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:30.045910 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b]
I20260926 01:56:30.048105 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90]
I20260926 01:56:30.053853 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c]
I20260926 01:56:30.061465 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c]
I20260926 01:56:30.062798 9341 tablet_bootstrap.cc:654] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:30.066704 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]
I20260926 01:56:30.072851 11157 raft_consensus.cc:2749] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:30.073408 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]
I20260926 01:56:30.075428 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1]
I20260926 01:56:30.083706 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad]
I20260926 01:56:30.089484 9152 leader_election.cc:304] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
W20260926 01:56:30.092859 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 67337.3us
I20260926 01:56:30.091094 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]
I20260926 01:56:30.096246 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d]
I20260926 01:56:30.099690 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]
I20260926 01:56:30.102712 10398 catalog_manager.cc:692] Loaded metadata for table default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4]
I20260926 01:56:30.122632 9341 tablet_bootstrap.cc:492] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:30.122777 9212 raft_consensus.cc:1240] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [term 2 LEADER]: Rejecting Update request from peer 49bd320bf73f4252804ec867ec077510 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260926 01:56:30.123145 9341 ts_tablet_manager.cc:1403] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.127s user 0.011s sys 0.004s
I20260926 01:56:30.125836 9341 raft_consensus.cc:359] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.127393 9341 raft_consensus.cc:385] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:30.127755 11057 consensus_queue.cc:1048] T ea75f309937e4ee49b35558117847b05 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:30.127884 9341 raft_consensus.cc:740] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:30.129096 9341 consensus_queue.cc:260] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.132424 9341 ts_tablet_manager.cc:1434] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260926 01:56:30.135594 9341 tablet_bootstrap.cc:492] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
W20260926 01:56:30.139319 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 49927.6us
I20260926 01:56:30.160243 9341 tablet_bootstrap.cc:654] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:30.173817 11003 consensus_queue.cc:1059] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:30.174840 11003 consensus_queue.cc:1243] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Peer bc00feec3f564608851f0112e97ab532 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260926 01:56:30.175561 11181 raft_consensus.cc:3055] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Stepping down as leader of term 1
I20260926 01:56:30.175873 11181 raft_consensus.cc:740] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49bd320bf73f4252804ec867ec077510, State: Running, Role: LEADER
I20260926 01:56:30.176671 11181 consensus_queue.cc:260] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.177587 11181 raft_consensus.cc:3060] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:30.203437 9068 raft_consensus.cc:1240] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Rejecting Update request from peer 49bd320bf73f4252804ec867ec077510 for earlier term 1. Current term is 2. Ops: []
I20260926 01:56:30.208822 10398 tablet_loader.cc:96] loaded metadata for tablet 030f4d683e404fbd8d2e6863e18f71c0 (table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72])
I20260926 01:56:30.219740 9341 tablet_bootstrap.cc:492] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:30.220247 9341 ts_tablet_manager.cc:1403] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.085s user 0.015s sys 0.002s
I20260926 01:56:30.225663 9341 raft_consensus.cc:359] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.231175 11157 raft_consensus.cc:2749] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:30.231541 9341 raft_consensus.cc:385] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:30.234097 9341 raft_consensus.cc:740] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:30.239574 9341 consensus_queue.cc:260] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.243362 10398 tablet_loader.cc:96] loaded metadata for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_57ad2cd7ce1e4bc6a071b8a0b985c3f2 [id=2af856781c904c77a11ca9f4966ca44b])
I20260926 01:56:30.253345 9341 ts_tablet_manager.cc:1434] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.033s user 0.006s sys 0.004s
I20260926 01:56:30.253859 10398 tablet_loader.cc:96] loaded metadata for tablet 05042bcc0a6e426d9e3e8b25d8d55540 (table default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469])
I20260926 01:56:30.254662 11157 raft_consensus.cc:493] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:30.255331 11157 raft_consensus.cc:515] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.257237 9341 tablet_bootstrap.cc:492] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:30.259594 9059 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:30.260391 9059 raft_consensus.cc:2393] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:30.261561 10398 tablet_loader.cc:96] loaded metadata for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f])
I20260926 01:56:30.262058 11157 leader_election.cc:290] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:30.268134 10398 tablet_loader.cc:96] loaded metadata for tablet 08b1d98089fd499a8e1ee217b561a801 (table default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0])
I20260926 01:56:30.272640 10398 tablet_loader.cc:96] loaded metadata for tablet 094b7485380b46fe8e1fef23ac50b830 (table default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181])
I20260926 01:56:30.279393 10398 tablet_loader.cc:96] loaded metadata for tablet 0ae1f842516445588886ed7e7149c02a (table default.Table_7d32da88077d4060913cc8ae86df83f6 [id=d2357d68ebb749358cb06a52c588b99f])
I20260926 01:56:30.280663 9341 tablet_bootstrap.cc:654] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:30.283207 10398 tablet_loader.cc:96] loaded metadata for tablet 0b2a99b3bfd64925a82236cec1d9e71f (table default.Table_05db3de2a0574423980de7d7b7098324 [id=5226bf119e6d4ffb839aea00e909e760])
I20260926 01:56:30.292315 10398 tablet_loader.cc:96] loaded metadata for tablet 0fdb7a7328c54031bd3afac1636d0f75 (table default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e])
I20260926 01:56:30.300627 10398 tablet_loader.cc:96] loaded metadata for tablet 1279a644e6604fb0bc20f53cd23dd8be (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a])
I20260926 01:56:30.304378 9341 tablet_bootstrap.cc:492] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:30.305203 9341 ts_tablet_manager.cc:1403] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.000s
I20260926 01:56:30.310087 10398 tablet_loader.cc:96] loaded metadata for tablet 14926a101b404f6b8d5a35b2aadc6c39 (table default.Table_bf92ccd2dbe1453d943f371ffec0badb [id=1cd337ec9ec94a47b24e774d0fff3321])
I20260926 01:56:30.313369 9341 raft_consensus.cc:359] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.315958 9341 raft_consensus.cc:385] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:30.317029 9341 raft_consensus.cc:740] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:30.319243 9341 consensus_queue.cc:260] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.326550 9341 ts_tablet_manager.cc:1434] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.021s user 0.007s sys 0.001s
I20260926 01:56:30.329104 10398 tablet_loader.cc:96] loaded metadata for tablet 1b84d9bc9c6f41bd80482c129e569648 (table default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b])
I20260926 01:56:30.330981 10398 tablet_loader.cc:96] loaded metadata for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c (table default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a])
I20260926 01:56:30.332899 10398 tablet_loader.cc:96] loaded metadata for tablet 1c5c66728bb54ec6beb4f0a2c25d9105 (table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c])
I20260926 01:56:30.334623 10398 tablet_loader.cc:96] loaded metadata for tablet 1c8761a6c3f7435193a91bfc4fe7fa86 (table default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8])
I20260926 01:56:30.338810 10398 tablet_loader.cc:96] loaded metadata for tablet 1f10cb68586b4a73abb86641b90c12f7 (table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f])
I20260926 01:56:30.343703 10398 tablet_loader.cc:96] loaded metadata for tablet 215a061440854c56b00df9fda9192598 (table default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e])
I20260926 01:56:30.345274 10398 tablet_loader.cc:96] loaded metadata for tablet 225bcc7bd4884188bba09b331b5da8e2 (table default.Table_0a43386150b14b53ad954d8f63768297 [id=27765f1075e54d509228a77011d6c8e8])
I20260926 01:56:30.346992 10398 tablet_loader.cc:96] loaded metadata for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a])
I20260926 01:56:30.350045 10398 tablet_loader.cc:96] loaded metadata for tablet 272bb87fe22c4720bcb3f0e07482dd8b (table default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1])
I20260926 01:56:30.352841 10398 tablet_loader.cc:96] loaded metadata for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409])
I20260926 01:56:30.354568 10398 tablet_loader.cc:96] loaded metadata for tablet 2a093fc10b604aa491b6e0f9596a589d (table default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e])
I20260926 01:56:30.356261 10398 tablet_loader.cc:96] loaded metadata for tablet 2c0cd3e30f084966b27921e658cacf58 (table default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146])
I20260926 01:56:30.362411 10398 tablet_loader.cc:96] loaded metadata for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd])
I20260926 01:56:30.365203 10398 tablet_loader.cc:96] loaded metadata for tablet 38a52e650df5449c86924387a0535d39 (table default.Table_e04a4278262e4d7f8d7a94ddf8666bbe [id=7879fa789a5143a6a1bb95768cf1bf50])
I20260926 01:56:30.367954 10398 tablet_loader.cc:96] loaded metadata for tablet 3a7f7eaf2d444886894b3b39902be4c9 (table default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d])
I20260926 01:56:30.372318 10398 tablet_loader.cc:96] loaded metadata for tablet 3c26f557ac274973824e6f9a4532de74 (table default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b])
I20260926 01:56:30.374634 10398 tablet_loader.cc:96] loaded metadata for tablet 3c7151ffe5d94a0599bbe3e6c8999371 (table default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c])
I20260926 01:56:30.375933 11201 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:30.376462 10398 tablet_loader.cc:96] loaded metadata for tablet 3f1d71caf78f4c75b144af14dce19d70 (table default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4])
I20260926 01:56:30.376519 11201 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:30.378268 10398 tablet_loader.cc:96] loaded metadata for tablet 4411120140f3410c9f4e33e26da20d98 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d])
I20260926 01:56:30.381817 10398 tablet_loader.cc:96] loaded metadata for tablet 4623456312284a698ae69d4054909647 (table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa])
I20260926 01:56:30.385601 10398 tablet_loader.cc:96] loaded metadata for tablet 4efbc1e1ef35420db51742d99d5a32bf (table default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410])
I20260926 01:56:30.387802 10398 tablet_loader.cc:96] loaded metadata for tablet 504cca4587d64bc4964b298ba1a74fe9 (table default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393])
I20260926 01:56:30.391191 10398 tablet_loader.cc:96] loaded metadata for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b])
I20260926 01:56:30.393277 10398 tablet_loader.cc:96] loaded metadata for tablet 56b0e40e5e964b90bd2dd158bbc4497a (table default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad])
I20260926 01:56:30.395172 10398 tablet_loader.cc:96] loaded metadata for tablet 572b97ad3aef4dce8cb3bcb521e90e48 (table default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 [id=30c67008e5de4b3da4aac5964c64c346])
I20260926 01:56:30.397116 11201 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:30.398324 10398 tablet_loader.cc:96] loaded metadata for tablet 5cbf646569e6497dac90e308f0681372 (table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b])
I20260926 01:56:30.398715 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:30.400404 10398 tablet_loader.cc:96] loaded metadata for tablet 5da0c41e2e19409d938f8f8ec29cb91a (table default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c])
I20260926 01:56:30.402442 10398 tablet_loader.cc:96] loaded metadata for tablet 5eef6a48e87e4c61a23058942a82bccc (table default.Table_4a4b59ee98dd4139b83d181413443621 [id=37400e3a991842198609512f0a9aeead])
I20260926 01:56:30.405128 10398 tablet_loader.cc:96] loaded metadata for tablet 5f300124bc3a4177a606b116bfef3a65 (table default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff])
I20260926 01:56:30.408828 10398 tablet_loader.cc:96] loaded metadata for tablet 661148692f9e4127af9abf5cc6cc0f54 (table default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1])
I20260926 01:56:30.411201 10398 tablet_loader.cc:96] loaded metadata for tablet 6629d125861f4e27b61c10d585a35c10 (table default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588])
I20260926 01:56:30.413766 10398 tablet_loader.cc:96] loaded metadata for tablet 6752df6a58914df5b8718dd0439aa885 (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63])
I20260926 01:56:30.416157 10398 tablet_loader.cc:96] loaded metadata for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2])
I20260926 01:56:30.418505 10398 tablet_loader.cc:96] loaded metadata for tablet 6ab082379974486bbea9f87f5aa64e10 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da])
I20260926 01:56:30.421828 10398 tablet_loader.cc:96] loaded metadata for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b (table default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b])
I20260926 01:56:30.426270 10398 tablet_loader.cc:96] loaded metadata for tablet 6f7419abcc2c482e97df70f03e047450 (table default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d])
I20260926 01:56:30.429955 10398 tablet_loader.cc:96] loaded metadata for tablet 70d2447335b34731a4cb2cd0f3b4bf13 (table default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e])
I20260926 01:56:30.431015 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a093fc10b604aa491b6e0f9596a589d" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:30.438104 10398 tablet_loader.cc:96] loaded metadata for tablet 7e57300d2bcf4e0496ed13a4e7d967f3 (table default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2])
I20260926 01:56:30.440444 10398 tablet_loader.cc:96] loaded metadata for tablet 812fe778734d4210b744ed8e45d412f1 (table default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe])
I20260926 01:56:30.442719 10398 tablet_loader.cc:96] loaded metadata for tablet 8711788a85f94c599224123f6b8cc814 (table default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c])
I20260926 01:56:30.444901 10398 tablet_loader.cc:96] loaded metadata for tablet 880d77d42c394e6a9cf0b3768a915402 (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4])
I20260926 01:56:30.447333 10398 tablet_loader.cc:96] loaded metadata for tablet 89fb965257de4db6b043b4a6dbe22709 (table default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc])
I20260926 01:56:30.453066 10398 tablet_loader.cc:96] loaded metadata for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791])
I20260926 01:56:30.456903 10398 tablet_loader.cc:96] loaded metadata for tablet 8aae2a23d3fe478e928757db4b30f5ca (table default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e])
I20260926 01:56:30.460632 10398 tablet_loader.cc:96] loaded metadata for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204])
I20260926 01:56:30.462896 10398 tablet_loader.cc:96] loaded metadata for tablet 91964074a37441db8b09f78720159586 (table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9])
I20260926 01:56:30.472172 10398 tablet_loader.cc:96] loaded metadata for tablet 9d973538ff534f05b301cb619475d4f3 (table default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a])
I20260926 01:56:30.475922 10398 tablet_loader.cc:96] loaded metadata for tablet 9db85241141f4a54a2b7dce15412a6f9 (table default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff])
I20260926 01:56:30.478228 10398 tablet_loader.cc:96] loaded metadata for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_85a98e90b8d2451d9b35f8bf4013c05b [id=008c87e2e12f48519efdacc69466442a])
I20260926 01:56:30.482052 10398 tablet_loader.cc:96] loaded metadata for tablet a6e8059b94564c42894c6c8a3d695950 (table default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c [id=d680dfae1d21415382be575ea6696a44])
I20260926 01:56:30.484025 10398 tablet_loader.cc:96] loaded metadata for tablet a73602570425443d97d28fcebe3cc6fc (table default.Table_818d89431f354c43ac77ba53fcb9cd8e [id=50846b080147404f91b349751918cca7])
I20260926 01:56:30.488750 10398 tablet_loader.cc:96] loaded metadata for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576])
I20260926 01:56:30.491055 10398 tablet_loader.cc:96] loaded metadata for tablet accc62f46ce24a4d97743dca05e65f51 (table default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7])
I20260926 01:56:30.493046 10398 tablet_loader.cc:96] loaded metadata for tablet af0840d6761a4521b0b6136b80eb6a6c (table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f])
I20260926 01:56:30.495318 10398 tablet_loader.cc:96] loaded metadata for tablet b25107dd5c104c83a2a21847bb2fc24f (table default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894])
I20260926 01:56:30.502024 10398 tablet_loader.cc:96] loaded metadata for tablet b778c95327fc477ca210cb3c983665c1 (table default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1])
I20260926 01:56:30.504292 10398 tablet_loader.cc:96] loaded metadata for tablet b85e2acbec2e485cb1b9728bd5763b0e (table default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264])
I20260926 01:56:30.513671 10398 tablet_loader.cc:96] loaded metadata for tablet c448caeeedfb4fc88eee88e40eebfeeb (table default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353])
I20260926 01:56:30.516006 10398 tablet_loader.cc:96] loaded metadata for tablet c450d101aa2b47c0bac3d4964c7bd6bd (table default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639])
I20260926 01:56:30.518321 10398 tablet_loader.cc:96] loaded metadata for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94])
I20260926 01:56:30.526628 11160 raft_consensus.cc:493] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:30.527127 11160 raft_consensus.cc:515] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:30.532747 11160 leader_election.cc:290] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:30.534502 9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:30.535414 9068 raft_consensus.cc:2393] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:30.552625 9152 leader_election.cc:304] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:30.584781 11157 raft_consensus.cc:3060] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:30.608829 10398 tablet_loader.cc:96] loaded metadata for tablet cdd0f4921da4436aa259486227a180d5 (table default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29])
W20260926 01:56:30.615870 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 63331.6us
I20260926 01:56:30.629359 10398 tablet_loader.cc:96] loaded metadata for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1])
I20260926 01:56:30.639892 11157 raft_consensus.cc:2749] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:30.649263 10398 tablet_loader.cc:96] loaded metadata for tablet d0f4e81776f942ff96a9c1f274996c6a (table default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96])
I20260926 01:56:30.655264 9214 raft_consensus.cc:3060] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:30.685671 9212 raft_consensus.cc:3060] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:30.691666 11204 consensus_queue.cc:1048] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:30.698787 8897 raft_consensus.cc:1275] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Refusing update from remote peer bc00feec3f564608851f0112e97ab532: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260926 01:56:30.699322 8897 pending_rounds.cc:85] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: Aborting all ops after (but not including) 0
I20260926 01:56:30.701450 8897 pending_rounds.cc:107] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260926 01:56:30.701831 8897 raft_consensus.cc:2889] T 70d2447335b34731a4cb2cd0f3b4bf13 P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260926 01:56:30.710664 10398 tablet_loader.cc:96] loaded metadata for tablet d7e2b22ca59745879270e8dc20f2a4fa (table default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d])
I20260926 01:56:30.723479 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08b1d98089fd499a8e1ee217b561a801" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:30.730337 10398 tablet_loader.cc:96] loaded metadata for tablet dbf222f9e14d4332b679f7ee94af1e0e (table default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6])
I20260926 01:56:30.745513 11204 consensus_queue.cc:1048] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:30.778007 10398 tablet_loader.cc:96] loaded metadata for tablet dd22a450f5914523871a088b24795e9f (table default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90])
I20260926 01:56:30.814198 9073 raft_consensus.cc:1275] T 70d2447335b34731a4cb2cd0f3b4bf13 P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Refusing update from remote peer bc00feec3f564608851f0112e97ab532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260926 01:56:30.817417 11160 consensus_queue.cc:1048] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260926 01:56:30.832566 10398 tablet_loader.cc:96] loaded metadata for tablet e140cf11b8bb49f9a94c0158a2afca4d (table default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83])
I20260926 01:56:30.839764 10398 tablet_loader.cc:96] loaded metadata for tablet e18ddaa6f0e748cf930e29cbca3844b1 (table default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b])
I20260926 01:56:30.892547 10398 tablet_loader.cc:96] loaded metadata for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9])
W20260926 01:56:30.909571 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 50598.4us
I20260926 01:56:30.903478 10398 tablet_loader.cc:96] loaded metadata for tablet e55a31bb9b9d4862b6bcef71df0cdaea (table default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932])
I20260926 01:56:30.927515 10398 tablet_loader.cc:96] loaded metadata for tablet e723112c1a6a4a6c89b6d17491d67c65 (table default.Table_da5237bb06194562a023680e328f087d [id=1c21bf55cfd54a3ca9fcb01d02491bfe])
I20260926 01:56:30.938557 11160 consensus_queue.cc:1048] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:30.944825 9152 leader_election.cc:304] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:30.945698 11160 raft_consensus.cc:2749] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:30.949976 10398 tablet_loader.cc:96] loaded metadata for tablet e7512e1d08e74ef6a2c699352347eb8d (table default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654])
I20260926 01:56:30.951390 10398 tablet_loader.cc:96] loaded metadata for tablet e76071c66b1d4635ac1a28110eb684fa (table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb])
I20260926 01:56:30.952629 10398 tablet_loader.cc:96] loaded metadata for tablet e85d17e9e48e4e54956e5c8460f7c9a8 (table default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e])
I20260926 01:56:30.953838 10398 tablet_loader.cc:96] loaded metadata for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7])
W20260926 01:56:30.974527 9910 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.593s (SENT) [suppressed 1 similar messages]
I20260926 01:56:30.974375 11210 consensus_queue.cc:1048] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:56:30.981669 10398 tablet_loader.cc:96] loaded metadata for tablet e8f5c3bdad494f34a3b72ab693599c55 (table default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc])
I20260926 01:56:30.987804 10398 tablet_loader.cc:96] loaded metadata for tablet ea75f309937e4ee49b35558117847b05 (table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4])
I20260926 01:56:30.989146 10398 tablet_loader.cc:96] loaded metadata for tablet ec749c375d784786ba93020996708bb1 (table default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6])
I20260926 01:56:30.990253 10398 tablet_loader.cc:96] loaded metadata for tablet edb7afa012944b3ba3c34bd994a95a83 (table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134])
I20260926 01:56:30.991515 10398 tablet_loader.cc:96] loaded metadata for tablet ee418347ebb04927913810d539ef57ea (table default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743])
I20260926 01:56:30.993468 10398 tablet_loader.cc:96] loaded metadata for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f])
I20260926 01:56:31.018565 10398 tablet_loader.cc:96] loaded metadata for tablet f2ee16b0c3e84936abba8248f194786a (table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c])
I20260926 01:56:31.021621 10398 tablet_loader.cc:96] loaded metadata for tablet f633f937b6c747f78efc6d3e167e83f0 (table default.Table_730735d338704c849d0e9a76c846e6cc [id=0e58ec4f07b143c3a4d1b9b6bd5735da])
I20260926 01:56:31.023087 11214 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:31.026376 11214 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:31.027631 10398 tablet_loader.cc:96] loaded metadata for tablet f74618f7c46744d2a08cb7b4e389c9f3 (table default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4])
I20260926 01:56:31.028777 10398 tablet_loader.cc:96] loaded metadata for tablet f7dcb2358e434f36a831352105863807 (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f])
I20260926 01:56:31.047803 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:31.051296 11214 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:31.071416 11160 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:31.072044 11160 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:31.075621 11160 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:31.076443 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:31.077258 9073 raft_consensus.cc:2393] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate d20f26c9295641ecbf77d290ca206bb2 in this term.
I20260926 01:56:31.080158 11160 raft_consensus.cc:493] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:31.080650 11160 raft_consensus.cc:515] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:31.096355 11160 leader_election.cc:290] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:31.102072 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 80915.4us
W20260926 01:56:31.110724 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 98443.2us
I20260926 01:56:31.114602 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
W20260926 01:56:31.116865 10398 catalog_manager.cc:1525] Time spent T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loading table and tablet metadata into memory: real 1.409s user 0.425s sys 0.086s
I20260926 01:56:31.117435 8539 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:42048 (request call id 336) took 1397 ms. Trace:
I20260926 01:56:31.117447 10398 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:56:31.117800 8539 rpcz_store.cc:276] 0926 01:56:29.720114 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:29.720241 (+ 127us) service_pool.cc:224] Handling call
0926 01:56:31.117361 (+1397120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:31.117651 8545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:31.119717 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:31.120990 8548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:31.121351 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
W20260926 01:56:31.121663 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 23) took 740 ms (client timeout 593 ms). Trace:
W20260926 01:56:31.121958 8548 rpcz_store.cc:269] 0926 01:56:30.380926 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:30.381052 (+ 126us) service_pool.cc:224] Handling call
0926 01:56:31.121594 (+740542us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:31.122884 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:31.123271 11214 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260926 01:56:31.130052 9155 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
I20260926 01:56:31.133450 8897 raft_consensus.cc:2393] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc00feec3f564608851f0112e97ab532 in current term 1: Already voted for candidate 49bd320bf73f4252804ec867ec077510 in this term.
I20260926 01:56:31.139914 11157 raft_consensus.cc:493] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:31.140435 11157 raft_consensus.cc:515] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:31.142354 11157 leader_election.cc:290] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:31.147415 9073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d20f26c9295641ecbf77d290ca206bb2" is_pre_election: true
I20260926 01:56:31.151343 11224 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260926 01:56:31.163036 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 46719.9us
W20260926 01:56:31.165643 9152 leader_election.cc:343] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 943a990fa10948e8961c02e9999569f5
I20260926 01:56:31.166148 9152 leader_election.cc:304] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:31.166903 11160 raft_consensus.cc:2749] T 943a990fa10948e8961c02e9999569f5 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:31.168015 9152 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:31.173696 11160 raft_consensus.cc:3060] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:31.199803 10398 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:56:31.200148 10398 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:56:31.204746 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" candidate_uuid: "bc00feec3f564608851f0112e97ab532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:31.218312 11160 raft_consensus.cc:2749] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:31.257153 9208 raft_consensus.cc:3060] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:56:31.280160 9152 leader_election.cc:304] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:31.366127 11134 raft_consensus.cc:2749] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:56:31.408257 10398 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:56:31.509999 11210 consensus_queue.cc:1048] T 08b1d98089fd499a8e1ee217b561a801 P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:31.510303 11209 consensus_queue.cc:1048] T b25107dd5c104c83a2a21847bb2fc24f P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:31.512091 11209 consensus_queue.cc:1048] T 2a093fc10b604aa491b6e0f9596a589d P 49bd320bf73f4252804ec867ec077510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:56:31.560441 8843 consensus_peers.cc:597] T 6d56f3a5dfed4ac78f1da92e22db558a P 49bd320bf73f4252804ec867ec077510 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d56f3a5dfed4ac78f1da92e22db558a. This is attempt 11: this message will repeat every 5th retry.
I20260926 01:56:31.591606 10398 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loaded TSK: 0
I20260926 01:56:31.607074 10398 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:56:31.752461 8539 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_f751bf0f1a19437aade46f4760f6f69b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:56:31.753383 8538 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_b1a3c70f85824c9a9d3e73d3f7544f05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:31.755082 8538 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1a3c70f85824c9a9d3e73d3f7544f05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:56:31.758322 8539 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f751bf0f1a19437aade46f4760f6f69b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:31.763866 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_619cd7cf639645f6a3039752fc75e5f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:31.765545 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_619cd7cf639645f6a3039752fc75e5f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:56:31.766554 9003 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 51792.8us
I20260926 01:56:31.773129 8537 master_service.cc:960] ReplaceTablet: received request to replace tablet 030f4d683e404fbd8d2e6863e18f71c0 from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:31.782963 8534 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8552d28bce1c484c964fe1268e370cc7" } modify_external_catalogs: true
I20260926 01:56:31.783613 8534 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8552d28bce1c484c964fe1268e370cc7" } modify_external_catalogs: true
I20260926 01:56:31.791302 8540 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8b014c6584dd427bbf5a612906ebb1b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:56:31.793046 8540 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b014c6584dd427bbf5a612906ebb1b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:31.801589 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 880d77d42c394e6a9cf0b3768a915402 from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:31.815579 9187 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e12827ad93c47cf9013e238215ceb7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:55:56 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:31.831176 11231 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:31.846086 11231 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:31.851971 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:31.852658 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f75ba46482de422ea23daf17aa100849 in term 9.
I20260926 01:56:31.854089 9923 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters:
I20260926 01:56:31.858825 11233 tablet_replica.cc:333] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:31.860203 11232 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:31.863683 11232 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:31.867040 11232 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:31.868216 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:31.868834 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef98e01cb9a44408ebf207163f208b5 in term 9.
I20260926 01:56:31.876857 11157 raft_consensus.cc:493] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d20f26c9295641ecbf77d290ca206bb2)
I20260926 01:56:31.877367 11157 raft_consensus.cc:515] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:31.879683 11157 leader_election.cc:290] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
W20260926 01:56:31.881973 9152 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
W20260926 01:56:31.883945 9155 leader_election.cc:343] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 1743e863cdd84f3f8ca413fa81cd1914
I20260926 01:56:31.870855 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, f75ba46482de422ea23daf17aa100849; no voters:
I20260926 01:56:31.884323 9155 leader_election.cc:304] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:31.885565 11134 raft_consensus.cc:2749] T 1743e863cdd84f3f8ca413fa81cd1914 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:31.890974 11232 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260926 01:56:31.893404 11232 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:31.894721 11232 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 9 FOLLOWER]: Advancing to term 10
I20260926 01:56:31.915481 11237 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260926 01:56:31.916028 11237 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:31.917227 11237 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 9 FOLLOWER]: Advancing to term 10
I20260926 01:56:31.921530 8548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:31.922479 11231 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:31.922912 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:31.931784 9923 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.189:34163, user_credentials={real_user=slave}} blocked reactor thread for 77886.8us
I20260926 01:56:31.937085 11232 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:31.939603 8624 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 10 FOLLOWER]: Rejecting Update request from peer ca25983598bc4c0688340d8e7bc0f6ba for earlier term 9. Current term is 10. Ops: [9.359-9.359]
I20260926 01:56:31.946787 11240 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: INVALID_TERM, Last received: 9.358, Next index: 359, Last known committed idx: 358, Time since last communication: 0.001s
I20260926 01:56:31.950769 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
I20260926 01:56:31.967242 11232 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:31.967969 9990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849"
I20260926 01:56:31.970613 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:31.973609 11232 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260926 01:56:31.986270 11236 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 9 LEADER]: Stepping down as leader of term 9
I20260926 01:56:31.987124 11236 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
I20260926 01:56:31.988986 11236 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 9.361, Last appended by leader: 361, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:31.990716 11236 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 9 FOLLOWER]: Advancing to term 10
I20260926 01:56:32.002820 11237 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:32.009167 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba"
I20260926 01:56:32.010869 9985 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 10 FOLLOWER]: Rejecting Update request from peer ca25983598bc4c0688340d8e7bc0f6ba for earlier term 9. Current term is 10. Ops: [9.359-9.359]
I20260926 01:56:32.018952 11233 raft_consensus.cc:2243] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:56:32.020078 11233 raft_consensus.cc:2272] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:32.022418 11233 ts_tablet_manager.cc:1916] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:31.953859 8536 catalog_manager.cc:5838] T ea75f309937e4ee49b35558117847b05 P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:32.035301 11222 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:32.035794 11222 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:32.038745 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532" is_pre_election: true
I20260926 01:56:32.039523 9212 raft_consensus.cc:2468] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:32.040962 11222 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:32.042448 8903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510" is_pre_election: true
I20260926 01:56:32.043100 8903 raft_consensus.cc:2468] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 1.
I20260926 01:56:32.031576 8536 catalog_manager.cc:5938] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395) does not have the latest schema for tablet ea75f309937e4ee49b35558117847b05 (table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]). Expected version 1 got 0
I20260926 01:56:32.046526 9003 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:32.048574 11178 raft_consensus.cc:2804] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:56:32.048929 11178 raft_consensus.cc:493] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:32.049283 11178 raft_consensus.cc:3060] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:32.059116 11237 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 election: Requested vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:32.048553 8536 catalog_manager.cc:5838] T d7e2b22ca59745879270e8dc20f2a4fa P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:32.060423 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 10 candidate_status { last_received { term: 9 index: 358 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:56:32.061229 8624 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f75ba46482de422ea23daf17aa100849 in current term 10: Already voted for candidate 9ef98e01cb9a44408ebf207163f208b5 in this term.
I20260926 01:56:32.060693 8536 catalog_manager.cc:5838] T 2a093fc10b604aa491b6e0f9596a589d P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:32.062068 8536 catalog_manager.cc:5838] T 08b1d98089fd499a8e1ee217b561a801 P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:32.063342 8536 catalog_manager.cc:5838] T b25107dd5c104c83a2a21847bb2fc24f P d20f26c9295641ecbf77d290ca206bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:32.063987 9923 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
W20260926 01:56:32.076190 8537 catalog_manager.cc:6899] Illegal state: an error occurred while writing to the sys-catalog: Replica ca25983598bc4c0688340d8e7bc0f6ba is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
W20260926 01:56:32.096617 8534 catalog_manager.cc:2887] Illegal state: an error occurred while updating the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.147s ago.
W20260926 01:56:32.099313 9275 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.76.190:44589)
W20260926 01:56:32.103124 9275 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.76.190:44589)
I20260926 01:56:32.114061 11233 ts_tablet_manager.cc:1929] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:32.114880 11233 log.cc:1199] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/2e12827ad93c47cf9013e238215ceb7a
W20260926 01:56:32.115459 8540 catalog_manager.cc:2241] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.166s ago.
I20260926 01:56:32.116003 11233 ts_tablet_manager.cc:1950] T 2e12827ad93c47cf9013e238215ceb7a P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
W20260926 01:56:32.126114 9275 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.190:44589)
I20260926 01:56:32.128015 11237 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260926 01:56:32.128670 8535 catalog_manager.cc:6899] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.171s ago.
I20260926 01:56:32.132416 11178 raft_consensus.cc:515] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
E20260926 01:56:32.115624 8536 catalog_manager.cc:6001] Error updating tablets from d20f26c9295641ecbf77d290ca206bb2: Illegal state: replica ca25983598bc4c0688340d8e7bc0f6ba is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "b25107dd5c104c83a2a21847bb2fc24f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "08b1d98089fd499a8e1ee217b561a801" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2a093fc10b604aa491b6e0f9596a589d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d7e2b22ca59745879270e8dc20f2a4fa" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ea75f309937e4ee49b35558117847b05" state: RUNNING consensus_state { current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } sequence_number: 107
I20260926 01:56:32.145493 9212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc00feec3f564608851f0112e97ab532"
I20260926 01:56:32.146080 9212 raft_consensus.cc:3060] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:56:32.146533 11178 leader_election.cc:290] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 election: Requested vote from peers bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045), 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257)
I20260926 01:56:32.273331 8897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aae2a23d3fe478e928757db4b30f5ca" candidate_uuid: "d20f26c9295641ecbf77d290ca206bb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bd320bf73f4252804ec867ec077510"
I20260926 01:56:32.273955 8897 raft_consensus.cc:3060] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:56:32.295406 10012 debug-util.cc:398] Leaking SignalData structure 0x7b080006f920 after lost signal to thread 9387
W20260926 01:56:32.296216 10012 debug-util.cc:398] Leaking SignalData structure 0x7b080006f2a0 after lost signal to thread 10042
I20260926 01:56:32.345373 9267 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
I20260926 01:56:32.372146 8897 raft_consensus.cc:2468] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 2.
I20260926 01:56:32.412249 9212 raft_consensus.cc:2468] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20f26c9295641ecbf77d290ca206bb2 in term 2.
I20260926 01:56:32.422829 9006 leader_election.cc:304] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2; no voters:
I20260926 01:56:32.425119 11221 raft_consensus.cc:2804] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:56:32.425669 11221 raft_consensus.cc:697] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [term 2 LEADER]: Becoming Leader. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Running, Role: LEADER
I20260926 01:56:32.426594 11221 consensus_queue.cc:237] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } }
I20260926 01:56:32.459781 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 2e12827ad93c47cf9013e238215ceb7a (table default.Table_df0e68c2c9ea42e18e91228cbf77f457 [id=639bcb9eb7674651b4a1d0e212648385]) successfully deleted
I20260926 01:56:32.523874 11236 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:32.524398 11236 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:32.528963 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 11 candidate_status { last_received { term: 9 index: 361 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:32.529211 11236 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:32.529580 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 10.
I20260926 01:56:32.529888 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 11 candidate_status { last_received { term: 9 index: 361 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:32.530493 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 10.
I20260926 01:56:32.534400 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:56:32.537516 11236 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260926 01:56:32.537928 11236 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:56:32.538286 11236 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 10 FOLLOWER]: Advancing to term 11
I20260926 01:56:32.553821 11236 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:32.588260 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 11 candidate_status { last_received { term: 9 index: 361 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5"
I20260926 01:56:32.588841 8624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 10 FOLLOWER]: Advancing to term 11
I20260926 01:56:32.596560 8624 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 11.
I20260926 01:56:32.597846 8526 leader_election.cc:304] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba; no voters:
I20260926 01:56:32.620380 9985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" candidate_term: 11 candidate_status { last_received { term: 9 index: 361 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849"
I20260926 01:56:32.620972 9985 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 10 FOLLOWER]: Advancing to term 11
I20260926 01:56:32.636173 9985 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca25983598bc4c0688340d8e7bc0f6ba in term 11.
I20260926 01:56:32.648753 11236 leader_election.cc:290] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [CANDIDATE]: Term 11 election: Requested vote from peers 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:32.680143 11236 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 11 FOLLOWER]: Leader election won for term 11
I20260926 01:56:32.684434 11236 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [term 11 LEADER]: Becoming Leader. State: Replica: ca25983598bc4c0688340d8e7bc0f6ba, State: Running, Role: LEADER
W20260926 01:56:32.682350 8526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.189:34163, user_credentials={real_user=slave}} blocked reactor thread for 84595us
I20260926 01:56:32.696228 11236 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 9.361, Last appended by leader: 361, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:32.745260 11250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 11 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:32.746340 11250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:33.154012 11134 raft_consensus.cc:493] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 49bd320bf73f4252804ec867ec077510)
I20260926 01:56:33.154593 11134 raft_consensus.cc:515] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:33.156723 11134 leader_election.cc:290] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
W20260926 01:56:33.164990 9152 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
W20260926 01:56:33.167665 9155 leader_election.cc:343] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 17b8b0d54f5645e5963f1de217f94c84
I20260926 01:56:33.168063 9155 leader_election.cc:304] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc00feec3f564608851f0112e97ab532; no voters: 49bd320bf73f4252804ec867ec077510, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:33.171372 11134 raft_consensus.cc:2749] T 17b8b0d54f5645e5963f1de217f94c84 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:56:33.196698 9985 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 9 index: 358. Preceding OpId from leader: term: 11 index: 362. (index mismatch)
I20260926 01:56:33.201197 11250 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 358, Time since last communication: 0.001s
I20260926 01:56:33.212778 8624 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Refusing update from remote peer ca25983598bc4c0688340d8e7bc0f6ba: Log matching property violated. Preceding OpId in replica: term: 9 index: 358. Preceding OpId from leader: term: 11 index: 362. (index mismatch)
I20260926 01:56:33.214619 11236 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 358, Time since last communication: 0.000s
W20260926 01:56:33.266219 9261 debug-util.cc:398] Leaking SignalData structure 0x7b08001409a0 after lost signal to thread 9130
W20260926 01:56:33.267666 9261 debug-util.cc:398] Leaking SignalData structure 0x7b0800112060 after lost signal to thread 9264
I20260926 01:56:33.348814 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 11 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:33.349742 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:33.462297 8538 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 354) took 1709 ms. Trace:
I20260926 01:56:33.462718 8538 rpcz_store.cc:276] 0926 01:56:31.752629 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:31.752763 (+ 134us) service_pool.cc:224] Handling call
0926 01:56:31.755515 (+ 2752us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:31.758758 (+ 3243us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:33.461557 (+1702799us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:33.461907 (+ 350us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:33.462218 (+ 311us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:31.811957 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:31.812730 (+ 773us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:31.812778 (+ 48us) write_op.cc:435] Acquired schema lock
0926 01:56:31.812813 (+ 35us) tablet.cc:662] Decoding operations
0926 01:56:31.813209 (+ 396us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:31.813315 (+ 106us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:31.813343 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:31.814603 (+ 1260us) tablet.cc:701] Row locks acquired
0926 01:56:31.814632 (+ 29us) write_op.cc:260] PREPARE: finished
0926 01:56:31.841480 (+ 26848us) write_op.cc:270] Start()
0926 01:56:31.841659 (+ 179us) write_op.cc:276] Timestamp: P: 1790387791841304 usec, L: 0
0926 01:56:31.841688 (+ 29us) op_driver.cc:348] REPLICATION: starting
0926 01:56:31.845021 (+ 3333us) log.cc:844] Serialized 433 byte log entry
0926 01:56:33.412886 (+1567865us) op_driver.cc:464] REPLICATION: finished
0926 01:56:33.446621 (+ 33735us) write_op.cc:301] APPLY: starting
0926 01:56:33.446703 (+ 82us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:33.446840 (+ 137us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:33.446961 (+ 121us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:33.447460 (+ 499us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:33.447671 (+ 211us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:33.447721 (+ 50us) write_op.cc:312] APPLY: finished
0926 01:56:33.447997 (+ 276us) log.cc:844] Serialized 29 byte log entry
0926 01:56:33.461140 (+ 13143us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:33.461231 (+ 91us) write_op.cc:454] Released schema lock
0926 01:56:33.461280 (+ 49us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33467,"apply.run_cpu_time_us":2085,"apply.run_wall_time_us":14977,"num_ops":2,"prepare.queue_time_us":48676,"prepare.run_cpu_time_us":25117,"prepare.run_wall_time_us":124869,"raft.queue_time_us":50979,"raft.run_cpu_time_us":29,"raft.run_wall_time_us":30,"replication_time_us":1571044,"thread_start_us":183234,"threads_started":5,"wal-append.queue_time_us":28557}]]}
I20260926 01:56:33.467774 8539 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 353) took 1715 ms. Trace:
I20260926 01:56:33.468832 8531 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:42048 (request call id 355) took 1706 ms. Trace:
I20260926 01:56:33.468142 8539 rpcz_store.cc:276] 0926 01:56:31.751895 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:31.752005 (+ 110us) service_pool.cc:224] Handling call
0926 01:56:31.758766 (+ 6761us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:31.760239 (+ 1473us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:33.467108 (+1706869us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:33.467420 (+ 312us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:33.467709 (+ 289us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:31.935257 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:31.935762 (+ 505us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:31.935787 (+ 25us) write_op.cc:435] Acquired schema lock
0926 01:56:31.935804 (+ 17us) tablet.cc:662] Decoding operations
0926 01:56:31.936392 (+ 588us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:31.936439 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:31.936464 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:31.936703 (+ 239us) tablet.cc:701] Row locks acquired
0926 01:56:31.936720 (+ 17us) write_op.cc:260] PREPARE: finished
0926 01:56:31.936867 (+ 147us) write_op.cc:270] Start()
0926 01:56:31.936970 (+ 103us) write_op.cc:276] Timestamp: P: 1790387791936778 usec, L: 0
0926 01:56:31.936984 (+ 14us) op_driver.cc:348] REPLICATION: starting
0926 01:56:31.937647 (+ 663us) log.cc:844] Serialized 433 byte log entry
0926 01:56:33.459609 (+1521962us) op_driver.cc:464] REPLICATION: finished
0926 01:56:33.461722 (+ 2113us) write_op.cc:301] APPLY: starting
0926 01:56:33.461775 (+ 53us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:33.461881 (+ 106us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:33.461985 (+ 104us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:33.462402 (+ 417us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:33.462579 (+ 177us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:33.462626 (+ 47us) write_op.cc:312] APPLY: finished
0926 01:56:33.462817 (+ 191us) log.cc:844] Serialized 29 byte log entry
0926 01:56:33.462970 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:33.463039 (+ 69us) write_op.cc:454] Released schema lock
0926 01:56:33.463073 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1961,"num_ops":2,"prepare.queue_time_us":173291,"prepare.run_cpu_time_us":2419,"prepare.run_wall_time_us":2641,"replication_time_us":1522493,"thread_start_us":4670,"threads_started":1}]]}
I20260926 01:56:33.469151 8531 rpcz_store.cc:276] 0926 01:56:31.762378 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:31.762526 (+ 148us) service_pool.cc:224] Handling call
0926 01:56:31.765925 (+ 3399us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:56:31.767666 (+ 1741us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:56:33.468203 (+1700537us) catalog_manager.cc:2246] Wrote table and tablets to system table
0926 01:56:33.468500 (+ 297us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0926 01:56:33.468761 (+ 261us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0926 01:56:31.938043 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0926 01:56:31.942499 (+ 4456us) write_op.cc:432] Acquiring schema lock in shared mode
0926 01:56:31.942539 (+ 40us) write_op.cc:435] Acquired schema lock
0926 01:56:31.942553 (+ 14us) tablet.cc:662] Decoding operations
0926 01:56:31.942908 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0926 01:56:31.942957 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0926 01:56:31.942984 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0926 01:56:31.943706 (+ 722us) tablet.cc:701] Row locks acquired
0926 01:56:31.943726 (+ 20us) write_op.cc:260] PREPARE: finished
0926 01:56:31.947602 (+ 3876us) write_op.cc:270] Start()
0926 01:56:31.948423 (+ 821us) write_op.cc:276] Timestamp: P: 1790387791947416 usec, L: 0
0926 01:56:31.948446 (+ 23us) op_driver.cc:348] REPLICATION: starting
0926 01:56:31.985731 (+ 37285us) log.cc:844] Serialized 433 byte log entry
0926 01:56:33.464645 (+1478914us) op_driver.cc:464] REPLICATION: finished
0926 01:56:33.465319 (+ 674us) write_op.cc:301] APPLY: starting
0926 01:56:33.465373 (+ 54us) tablet.cc:1394] starting BulkCheckPresence
0926 01:56:33.465480 (+ 107us) tablet.cc:1397] finished BulkCheckPresence
0926 01:56:33.465601 (+ 121us) tablet.cc:1399] starting ApplyRowOperation cycle
0926 01:56:33.466048 (+ 447us) tablet.cc:1410] finished ApplyRowOperation cycle
0926 01:56:33.466203 (+ 155us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0926 01:56:33.466243 (+ 40us) write_op.cc:312] APPLY: finished
0926 01:56:33.466485 (+ 242us) log.cc:844] Serialized 29 byte log entry
0926 01:56:33.466614 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0926 01:56:33.466682 (+ 68us) write_op.cc:454] Released schema lock
0926 01:56:33.466712 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":470,"apply.run_cpu_time_us":1628,"apply.run_wall_time_us":1628,"num_ops":2,"prepare.queue_time_us":169039,"prepare.run_cpu_time_us":3240,"prepare.run_wall_time_us":48293,"raft.queue_time_us":6695,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":12,"replication_time_us":1516109,"spinlock_wait_cycles":1227648}]]}
I20260926 01:56:33.484969 8498 master-stress-test.cc:581] Killed test thread 1/10
I20260926 01:56:33.485284 8498 master-stress-test.cc:579] Killing test thread 2/10
I20260926 01:56:33.493645 11236 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:33.494424 11236 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:33.505993 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:33.506752 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:33.531394 11252 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:33.532423 11252 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:33.533136 9216 raft_consensus.cc:1275] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Refusing update from remote peer d20f26c9295641ecbf77d290ca206bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260926 01:56:33.534904 11178 consensus_queue.cc:1048] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:33.539460 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca25983598bc4c0688340d8e7bc0f6ba. Latest consensus state: current_term: 11 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:33.541843 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:56:33.548482 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:33.584293 11252 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:33.584481 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:33.586373 9923 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:33.594960 11250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:33.595777 11250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba [sys.catalog]: This master's current role is: LEADER
I20260926 01:56:33.621547 11251 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:56:33.636960 8897 raft_consensus.cc:1275] T 8aae2a23d3fe478e928757db4b30f5ca P 49bd320bf73f4252804ec867ec077510 [term 2 FOLLOWER]: Refusing update from remote peer d20f26c9295641ecbf77d290ca206bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260926 01:56:33.640201 11221 consensus_queue.cc:1048] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:56:33.670419 11252 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260926 01:56:33.672043 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } } }
I20260926 01:56:33.672683 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:56:33.732038 11276 log.cc:927] Time spent T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260926 01:56:33.773247 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_85a98e90b8d2451d9b35f8bf4013c05b [id=008c87e2e12f48519efdacc69466442a]
I20260926 01:56:33.779865 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f]
I20260926 01:56:33.794946 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409]
I20260926 01:56:33.830773 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]
I20260926 01:56:33.886639 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_730735d338704c849d0e9a76c846e6cc [id=0e58ec4f07b143c3a4d1b9b6bd5735da]
I20260926 01:56:33.904253 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]
I20260926 01:56:33.919288 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469]
I20260926 01:56:33.933518 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e]
W20260926 01:56:33.996171 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.130:41395, user_credentials={real_user=slave}} blocked reactor thread for 48716.5us
I20260926 01:56:34.001600 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94]
I20260926 01:56:34.011276 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c]
I20260926 01:56:34.030714 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d]
I20260926 01:56:34.041970 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0]
I20260926 01:56:34.060413 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_da5237bb06194562a023680e328f087d [id=1c21bf55cfd54a3ca9fcb01d02491bfe]
I20260926 01:56:34.071539 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_bf92ccd2dbe1453d943f371ffec0badb [id=1cd337ec9ec94a47b24e774d0fff3321]
I20260926 01:56:34.075903 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c]
I20260926 01:56:34.089483 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b]
I20260926 01:56:34.095010 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa]
I20260926 01:56:34.128031 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_b1a3c70f85824c9a9d3e73d3f7544f05 [id=21ac5392a7f64b5ea1052d99a885a2bf]
I20260926 01:56:34.129756 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743]
I20260926 01:56:34.133100 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_0a43386150b14b53ad954d8f63768297 [id=27765f1075e54d509228a77011d6c8e8]
I20260926 01:56:34.141170 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e]
I20260926 01:56:34.142706 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181]
I20260926 01:56:34.170655 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_57ad2cd7ce1e4bc6a071b8a0b985c3f2 [id=2af856781c904c77a11ca9f4966ca44b]
I20260926 01:56:34.173039 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f]
I20260926 01:56:34.178074 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1]
I20260926 01:56:34.188534 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]
I20260926 01:56:34.195731 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb]
I20260926 01:56:34.203846 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 [id=30c67008e5de4b3da4aac5964c64c346]
I20260926 01:56:34.208339 11282 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:34.209120 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e]
I20260926 01:56:34.213191 11282 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:34.217886 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6]
I20260926 01:56:34.220945 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4a4b59ee98dd4139b83d181413443621 [id=37400e3a991842198609512f0a9aeead]
I20260926 01:56:34.222990 11282 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:34.224071 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:34.228878 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8]
I20260926 01:56:34.240928 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff]
I20260926 01:56:34.251964 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e]
I20260926 01:56:34.254091 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410]
I20260926 01:56:34.256898 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe]
I20260926 01:56:34.258159 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932]
I20260926 01:56:34.305011 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7]
I20260926 01:56:34.320053 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4]
I20260926 01:56:34.321616 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_818d89431f354c43ac77ba53fcb9cd8e [id=50846b080147404f91b349751918cca7]
I20260926 01:56:34.322981 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_05db3de2a0574423980de7d7b7098324 [id=5226bf119e6d4ffb839aea00e909e760]
I20260926 01:56:34.346120 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791]
I20260926 01:56:34.368068 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a]
I20260926 01:56:34.391913 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1]
I20260926 01:56:34.401844 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d]
I20260926 01:56:34.403409 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc]
I20260926 01:56:34.436990 11283 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:34.437517 11283 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:34.450940 11277 raft_consensus.cc:493] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc00feec3f564608851f0112e97ab532)
I20260926 01:56:34.451610 11277 raft_consensus.cc:515] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } }
I20260926 01:56:34.453946 11277 leader_election.cc:290] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395), bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045)
W20260926 01:56:34.456940 8843 leader_election.cc:343] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): Not found: Tablet not found: 2e12827ad93c47cf9013e238215ceb7a
W20260926 01:56:34.459959 8843 leader_election.cc:343] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Not found: Tablet not found: 2e12827ad93c47cf9013e238215ceb7a
I20260926 01:56:34.460333 8843 leader_election.cc:304] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49bd320bf73f4252804ec867ec077510; no voters: bc00feec3f564608851f0112e97ab532, d20f26c9295641ecbf77d290ca206bb2
I20260926 01:56:34.461328 11277 raft_consensus.cc:2749] T 2e12827ad93c47cf9013e238215ceb7a P 49bd320bf73f4252804ec867ec077510 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:56:34.450084 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 50047.4us
I20260926 01:56:34.490985 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393]
I20260926 01:56:34.509580 9990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:34.511747 11283 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:34.575533 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7]
I20260926 01:56:34.578495 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353]
I20260926 01:56:34.655619 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]
I20260926 01:56:34.670986 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_e04a4278262e4d7f8d7a94ddf8666bbe [id=7879fa789a5143a6a1bb95768cf1bf50]
I20260926 01:56:34.691156 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd]
I20260926 01:56:34.699492 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264]
W20260926 01:56:34.714447 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 53058.7us
I20260926 01:56:34.706240 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576]
I20260926 01:56:34.721019 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2]
I20260926 01:56:34.750348 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588]
I20260926 01:56:34.759786 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]
I20260926 01:56:34.768281 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b]
I20260926 01:56:34.783437 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654]
I20260926 01:56:34.788652 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6]
I20260926 01:56:34.804232 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f]
I20260926 01:56:34.806490 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]
I20260926 01:56:34.808354 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e]
I20260926 01:56:34.809465 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83]
I20260926 01:56:34.839025 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29]
I20260926 01:56:34.847499 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b]
I20260926 01:56:34.867450 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a]
I20260926 01:56:34.869580 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d]
I20260926 01:56:34.910729 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b]
I20260926 01:56:34.932866 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e]
I20260926 01:56:34.934114 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]
I20260926 01:56:34.936566 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_619cd7cf639645f6a3039752fc75e5f6 [id=beae83b06ad14413b3ecff25b28af923]
W20260926 01:56:34.948839 8599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.508s (SENT) [suppressed 1 similar messages]
I20260926 01:56:34.959010 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1]
I20260926 01:56:34.960676 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]
I20260926 01:56:34.987612 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9]
I20260926 01:56:34.998338 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639]
I20260926 01:56:34.999857 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204]
W20260926 01:56:35.023303 8599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.76.190:44589, user_credentials={real_user=slave}} blocked reactor thread for 74383.6us
I20260926 01:56:35.050644 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_7d32da88077d4060913cc8ae86df83f6 [id=d2357d68ebb749358cb06a52c588b99f]
I20260926 01:56:35.058713 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c [id=d680dfae1d21415382be575ea6696a44]
I20260926 01:56:35.064107 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96]
I20260926 01:56:35.073925 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff]
I20260926 01:56:35.079636 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_f751bf0f1a19437aade46f4760f6f69b [id=d8ddf10a94d3432b9fd6eac6f6c27e95]
I20260926 01:56:35.085803 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]
I20260926 01:56:35.092664 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146]
I20260926 01:56:35.097477 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894]
I20260926 01:56:35.102442 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b]
I20260926 01:56:35.108500 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90]
I20260926 01:56:35.117412 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c]
I20260926 01:56:35.124993 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c]
I20260926 01:56:35.126868 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]
I20260926 01:56:35.132300 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a]
I20260926 01:56:35.138191 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1]
I20260926 01:56:35.145222 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad]
I20260926 01:56:35.147903 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c]
I20260926 01:56:35.158346 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d]
I20260926 01:56:35.160347 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9]
I20260926 01:56:35.162147 11251 catalog_manager.cc:692] Loaded metadata for table default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4]
I20260926 01:56:35.277202 11251 tablet_loader.cc:96] loaded metadata for tablet 030f4d683e404fbd8d2e6863e18f71c0 (table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72])
I20260926 01:56:35.279390 11251 tablet_loader.cc:96] loaded metadata for tablet 04e79a2e52fb40619b2bc99396abda5b (table default.Table_57ad2cd7ce1e4bc6a071b8a0b985c3f2 [id=2af856781c904c77a11ca9f4966ca44b])
I20260926 01:56:35.280539 11251 tablet_loader.cc:96] loaded metadata for tablet 05042bcc0a6e426d9e3e8b25d8d55540 (table default.Table_db38e199ec7b405f87d487e2ff6d24f5 [id=11972c44eb76454d973e5be26468c469])
I20260926 01:56:35.282341 11251 tablet_loader.cc:96] loaded metadata for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f])
I20260926 01:56:35.284037 11251 tablet_loader.cc:96] loaded metadata for tablet 08b1d98089fd499a8e1ee217b561a801 (table default.Table_2bec71bcc14b4425a9ed67b1e1adf9bc [id=1c02fdab21fd4a2b81c36b944f5bead0])
I20260926 01:56:35.285496 11251 tablet_loader.cc:96] loaded metadata for tablet 094b7485380b46fe8e1fef23ac50b830 (table default.Table_4de0847b69e04e49b345919625be76a6 [id=297861b136ae47d5bd596654c3faa181])
I20260926 01:56:35.287514 11251 tablet_loader.cc:96] loaded metadata for tablet 0ae1f842516445588886ed7e7149c02a (table default.Table_7d32da88077d4060913cc8ae86df83f6 [id=d2357d68ebb749358cb06a52c588b99f])
I20260926 01:56:35.288412 11251 tablet_loader.cc:96] loaded metadata for tablet 0b2a99b3bfd64925a82236cec1d9e71f (table default.Table_05db3de2a0574423980de7d7b7098324 [id=5226bf119e6d4ffb839aea00e909e760])
I20260926 01:56:35.291378 11251 tablet_loader.cc:96] loaded metadata for tablet 0fdb7a7328c54031bd3afac1636d0f75 (table default.Table_011a9ce871e74c2fa862dd3b85e33c32 [id=b9858a721af344ebb80e801becfcc08e])
I20260926 01:56:35.311192 11251 tablet_loader.cc:96] loaded metadata for tablet 1279a644e6604fb0bc20f53cd23dd8be (table default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a])
I20260926 01:56:35.334689 11251 tablet_loader.cc:96] loaded metadata for tablet 14926a101b404f6b8d5a35b2aadc6c39 (table default.Table_bf92ccd2dbe1453d943f371ffec0badb [id=1cd337ec9ec94a47b24e774d0fff3321])
I20260926 01:56:35.348122 11251 tablet_loader.cc:96] loaded metadata for tablet 1b84d9bc9c6f41bd80482c129e569648 (table default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b])
I20260926 01:56:35.395713 11251 tablet_loader.cc:96] loaded metadata for tablet 1c1d70856f5c4ac68a5fcc8b5e1ef89c (table default.Table_31abd87321d54403b8bbd41d2d7c8dc4 [id=5a0dd0aa5a9a4bb39c37130743a5951a])
I20260926 01:56:35.405893 11251 tablet_loader.cc:96] loaded metadata for tablet 1c5c66728bb54ec6beb4f0a2c25d9105 (table default.Table_1c3a8d6844ce4b98b9337d97f0ff1cf4 [id=f3588cbc884447cf86cf68b4c99bdd2c])
I20260926 01:56:35.407300 11251 tablet_loader.cc:96] loaded metadata for tablet 1c8761a6c3f7435193a91bfc4fe7fa86 (table default.Table_a21bbc98baa64f09a86dde0520e49cb9 [id=39f370e40fb74f7db3daf890ffc1d5b8])
I20260926 01:56:35.410269 11251 tablet_loader.cc:96] loaded metadata for tablet 1f10cb68586b4a73abb86641b90c12f7 (table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f])
I20260926 01:56:35.413794 11251 tablet_loader.cc:96] loaded metadata for tablet 215a061440854c56b00df9fda9192598 (table default.Table_776639f9efaf43db9a1ca82994fe91b0 [id=27f31069e1ef43d2b301ec803dac728e])
I20260926 01:56:35.417215 11251 tablet_loader.cc:96] loaded metadata for tablet 225bcc7bd4884188bba09b331b5da8e2 (table default.Table_0a43386150b14b53ad954d8f63768297 [id=27765f1075e54d509228a77011d6c8e8])
I20260926 01:56:35.418630 11251 tablet_loader.cc:96] loaded metadata for tablet 229f4cdd841949e4a12e8555cdfe42cc (table default.Table_3e325156f7684b6e991c8cb0ed626c25 [id=e7f06f7838a740a09da1cfda5d27fc2a])
I20260926 01:56:35.421788 11251 tablet_loader.cc:96] loaded metadata for tablet 272bb87fe22c4720bcb3f0e07482dd8b (table default.Table_c74d34ecd1ef4a6d8989b1017deca342 [id=ea31230b8b584fd9bf166477243462d1])
I20260926 01:56:35.424108 11251 tablet_loader.cc:96] loaded metadata for tablet 28eaf7381f364a2682dd07f2bd90ede3 (table default.Table_7fe57989dc2540bca5e9adf318b41015 [id=08a05d07365148b0b9e246f3fcc6d409])
I20260926 01:56:35.426373 11251 tablet_loader.cc:96] loaded metadata for tablet 2a093fc10b604aa491b6e0f9596a589d (table default.Table_6e853298109b4319a558da622f72f21b [id=9bdeb13d2295425dadcbacc76f4a995e])
I20260926 01:56:35.427675 11251 tablet_loader.cc:96] loaded metadata for tablet 2c0cd3e30f084966b27921e658cacf58 (table default.Table_59311caf53c74458b4c17a7907ed0afb [id=dacf73f537dc4109b759f36678ea8146])
I20260926 01:56:35.432130 11251 tablet_loader.cc:96] loaded metadata for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd])
I20260926 01:56:35.434088 11251 tablet_loader.cc:96] loaded metadata for tablet 38a52e650df5449c86924387a0535d39 (table default.Table_e04a4278262e4d7f8d7a94ddf8666bbe [id=7879fa789a5143a6a1bb95768cf1bf50])
I20260926 01:56:35.436286 11251 tablet_loader.cc:96] loaded metadata for tablet 3a7f7eaf2d444886894b3b39902be4c9 (table default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d])
I20260926 01:56:35.439108 11251 tablet_loader.cc:96] loaded metadata for tablet 3c26f557ac274973824e6f9a4532de74 (table default.Table_cef381857b034f04a16cd5bbd1f20eb3 [id=874843625439432fac61d973abf8007b])
I20260926 01:56:35.440291 11251 tablet_loader.cc:96] loaded metadata for tablet 3c7151ffe5d94a0599bbe3e6c8999371 (table default.Table_23c61cea9e0648838bc80efc34b50799 [id=df1b0511560f4fbbb7a48f5f1309346c])
I20260926 01:56:35.486256 11251 tablet_loader.cc:96] loaded metadata for tablet 3f1d71caf78f4c75b144af14dce19d70 (table default.Table_e1b0b5fe9a4b49c5859b8d3b6ab7adde [id=507fc8b6c11b4ac183e3188d72eeced4])
I20260926 01:56:35.487664 11251 tablet_loader.cc:96] loaded metadata for tablet 4411120140f3410c9f4e33e26da20d98 (table default.Table_e8fc0c2764f3477c97065d16cab079b3 [id=1a453fa859664e949b7fcbbcaef0f74d])
I20260926 01:56:35.489537 11251 tablet_loader.cc:96] loaded metadata for tablet 4623456312284a698ae69d4054909647 (table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa])
I20260926 01:56:35.527658 11251 tablet_loader.cc:96] loaded metadata for tablet 4efbc1e1ef35420db51742d99d5a32bf (table default.Table_4e4c17faa0a4400d969f7de29bf7dcee [id=45a48a94e6ff46fe8b177d6b78295410])
W20260926 01:56:35.534042 8599 leader_election.cc:336] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.508s (SENT)
I20260926 01:56:35.534520 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:35.547204 11293 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260926 01:56:35.548643 11251 tablet_loader.cc:96] loaded metadata for tablet 504cca4587d64bc4964b298ba1a74fe9 (table default.Table_49bdcbf4ba8b4771b549ee062065c8e5 [id=63c7ce8fbc18424a80b37fa503d6a393])
I20260926 01:56:35.566790 11251 tablet_loader.cc:96] loaded metadata for tablet 562604137d484e83a51de2f778d58fca (table default.Table_a8ab0ff60e564bac9e08a9a1c7736a10 [id=b202eb85ec1944d8ad57f8521b37170b])
I20260926 01:56:35.611244 11251 tablet_loader.cc:96] loaded metadata for tablet 56b0e40e5e964b90bd2dd158bbc4497a (table default.Table_1e95ea0e55dd4033ae92dde6bb3567de [id=f1512e6c23554e46b7af14f8771a17ad])
I20260926 01:56:35.632346 11251 tablet_loader.cc:96] loaded metadata for tablet 572b97ad3aef4dce8cb3bcb521e90e48 (table default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 [id=30c67008e5de4b3da4aac5964c64c346])
I20260926 01:56:35.638139 11251 tablet_loader.cc:96] loaded metadata for tablet 5cbf646569e6497dac90e308f0681372 (table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b])
I20260926 01:56:35.640152 11251 tablet_loader.cc:96] loaded metadata for tablet 5da0c41e2e19409d938f8f8ec29cb91a (table default.Table_56df0fbe0b8949b4b6792b98d7fed21a [id=e45180175c6642cea9464145743be11c])
I20260926 01:56:35.642000 11251 tablet_loader.cc:96] loaded metadata for tablet 5eef6a48e87e4c61a23058942a82bccc (table default.Table_4a4b59ee98dd4139b83d181413443621 [id=37400e3a991842198609512f0a9aeead])
I20260926 01:56:35.643882 11251 tablet_loader.cc:96] loaded metadata for tablet 5f300124bc3a4177a606b116bfef3a65 (table default.Table_4c909c1cd0a54659b097c9ef7f487867 [id=3cc257f5123c492b9c21ae0cb7983aff])
I20260926 01:56:35.647037 11251 tablet_loader.cc:96] loaded metadata for tablet 661148692f9e4127af9abf5cc6cc0f54 (table default.Table_b205886d445546d08910c2f6fb5db3c4 [id=2d591ac37c504d798e0ae8387c03a0d1])
I20260926 01:56:35.662310 11251 tablet_loader.cc:96] loaded metadata for tablet 6629d125861f4e27b61c10d585a35c10 (table default.Table_63324ad934044895b71a28a0e5286b8b [id=851708598ad1466f97eaaf1131030588])
I20260926 01:56:35.667898 11251 tablet_loader.cc:96] loaded metadata for tablet 6752df6a58914df5b8718dd0439aa885 (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63])
I20260926 01:56:35.670009 11251 tablet_loader.cc:96] loaded metadata for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2])
I20260926 01:56:35.672143 11251 tablet_loader.cc:96] loaded metadata for tablet 6ab082379974486bbea9f87f5aa64e10 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da])
I20260926 01:56:35.675006 11251 tablet_loader.cc:96] loaded metadata for tablet 6cbfa4fa8ab146108cd2c62a0de90b40 (table default.Table_f751bf0f1a19437aade46f4760f6f69b [id=d8ddf10a94d3432b9fd6eac6f6c27e95])
I20260926 01:56:35.677073 11251 tablet_loader.cc:96] loaded metadata for tablet 6d101ad03f8e4d7f8bf70ea993c4f58b (table default.Table_cc558351862b4a39bc56a3883cba12ce [id=a2140f8cd1d443b4a561621292f4116b])
I20260926 01:56:35.681692 11251 tablet_loader.cc:96] loaded metadata for tablet 6f7419abcc2c482e97df70f03e047450 (table default.Table_2bcec3148a5d418198f3c1adb758b949 [id=5ed5fe49e23f4b47ab8ee73cee582a2d])
I20260926 01:56:35.684932 11251 tablet_loader.cc:96] loaded metadata for tablet 70d2447335b34731a4cb2cd0f3b4bf13 (table default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e])
I20260926 01:56:35.692243 11251 tablet_loader.cc:96] loaded metadata for tablet 7e57300d2bcf4e0496ed13a4e7d967f3 (table default.Table_68c0da73fcc54bf99603db86570ccbe6 [id=849e08c0a84e4b1da649b47f48bfe1e2])
I20260926 01:56:35.694420 11251 tablet_loader.cc:96] loaded metadata for tablet 812fe778734d4210b744ed8e45d412f1 (table default.Table_93d21537b7c149098fb8fd3a5120fd27 [id=48c04b483c57400a9da3b4ee720e95fe])
I20260926 01:56:35.696704 11251 tablet_loader.cc:96] loaded metadata for tablet 8711788a85f94c599224123f6b8cc814 (table default.Table_b644d2630c07480e8cc4ec135c1b91f8 [id=1e5f85b22c3848949f3c55afdff1bd5c])
I20260926 01:56:35.698691 11251 tablet_loader.cc:96] loaded metadata for tablet 880d77d42c394e6a9cf0b3768a915402 (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4])
I20260926 01:56:35.700578 11251 tablet_loader.cc:96] loaded metadata for tablet 89fb965257de4db6b043b4a6dbe22709 (table default.Table_ee8c0a5c849548228d58b9d2bb0c47ec [id=5ee702f1de4e41b49bc85e63214158bc])
I20260926 01:56:35.705024 11251 tablet_loader.cc:96] loaded metadata for tablet 8a56044249bb4aaa81903d2a8aeb008e (table default.Table_1871a77d01894075a74872c5af9622ec [id=594b98551ce143769bbada50ba44a791])
I20260926 01:56:35.706918 11251 tablet_loader.cc:96] loaded metadata for tablet 8aae2a23d3fe478e928757db4b30f5ca (table default.Table_519e634e0d394eeba89b9eea66db9bd6 [id=11c12396dc03440db6de7779cb47088e])
I20260926 01:56:35.709947 11251 tablet_loader.cc:96] loaded metadata for tablet 8dfff96cc29145e7a72213bb5ecca1ae (table default.Table_e313053803ad4a608bc4a5a546458c5d [id=c8194b05bc344ec88191d0ae47dd3204])
I20260926 01:56:35.711849 11251 tablet_loader.cc:96] loaded metadata for tablet 91964074a37441db8b09f78720159586 (table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9])
I20260926 01:56:35.719692 11251 tablet_loader.cc:96] loaded metadata for tablet 9d973538ff534f05b301cb619475d4f3 (table default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a])
I20260926 01:56:35.722769 11251 tablet_loader.cc:96] loaded metadata for tablet 9db85241141f4a54a2b7dce15412a6f9 (table default.Table_83505dff1c1149ebb8e82a18792b4fa3 [id=d7289e88aa10461389d050c48af79dff])
I20260926 01:56:35.724732 11251 tablet_loader.cc:96] loaded metadata for tablet a09351c3db384b4383ca5ff0eaca982c (table default.Table_85a98e90b8d2451d9b35f8bf4013c05b [id=008c87e2e12f48519efdacc69466442a])
I20260926 01:56:35.727883 11251 tablet_loader.cc:96] loaded metadata for tablet a6e8059b94564c42894c6c8a3d695950 (table default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c [id=d680dfae1d21415382be575ea6696a44])
I20260926 01:56:35.729470 11251 tablet_loader.cc:96] loaded metadata for tablet a73602570425443d97d28fcebe3cc6fc (table default.Table_818d89431f354c43ac77ba53fcb9cd8e [id=50846b080147404f91b349751918cca7])
I20260926 01:56:35.735631 11251 tablet_loader.cc:96] loaded metadata for tablet ac2e6e544aae4ac3a2d3cc116d22e581 (table default.Table_e96413b8ff0340f9b3698862eb4525fe [id=807729d588d64cf58ce3ec823a616576])
I20260926 01:56:35.737627 11251 tablet_loader.cc:96] loaded metadata for tablet accc62f46ce24a4d97743dca05e65f51 (table default.Table_a8580d0d4cfe42c594511d26fbcff4b4 [id=4fe742afbad64390b21c8599a22740d7])
I20260926 01:56:35.739287 11251 tablet_loader.cc:96] loaded metadata for tablet af0840d6761a4521b0b6136b80eb6a6c (table default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f])
I20260926 01:56:35.741154 11251 tablet_loader.cc:96] loaded metadata for tablet b25107dd5c104c83a2a21847bb2fc24f (table default.Table_dc2d02d1ff49427d81cc4c14d1f952c7 [id=dae2efa01e5c47988642ff40deab6894])
I20260926 01:56:35.746685 11251 tablet_loader.cc:96] loaded metadata for tablet b778c95327fc477ca210cb3c983665c1 (table default.Table_851c22c2b5c04d179232ac610720efd9 [id=bfe5f854de1448fe97b77aa639dd1ad1])
I20260926 01:56:35.749676 11251 tablet_loader.cc:96] loaded metadata for tablet b85e2acbec2e485cb1b9728bd5763b0e (table default.Table_a97d7498cf3d4cf18177932f26b3a1db [id=7f4d0a7a11cd4a56b4a61ce03d3d3264])
I20260926 01:56:35.753554 11251 tablet_loader.cc:96] loaded metadata for tablet bce97ff01117487a8fee85b5349679a2 (table default.Table_619cd7cf639645f6a3039752fc75e5f6 [id=beae83b06ad14413b3ecff25b28af923])
W20260926 01:56:35.756515 9114 debug-util.cc:398] Leaking SignalData structure 0x7b0800197760 after lost signal to thread 8967
W20260926 01:56:35.757872 9114 debug-util.cc:398] Leaking SignalData structure 0x7b0800197b40 after lost signal to thread 9117
I20260926 01:56:35.759887 11251 tablet_loader.cc:96] loaded metadata for tablet c448caeeedfb4fc88eee88e40eebfeeb (table default.Table_7f0d970c20fe4172b726678f7ddcb661 [id=6ad3d5d0571d4d5da2d236d904784353])
I20260926 01:56:35.761745 11251 tablet_loader.cc:96] loaded metadata for tablet c450d101aa2b47c0bac3d4964c7bd6bd (table default.Table_8b2627427be143419dd9a44285379cd3 [id=c611ebdee9a84ea1baa3a7b5d010c639])
I20260926 01:56:35.763622 11251 tablet_loader.cc:96] loaded metadata for tablet c9f54f648a55404fb6622b35d84368ee (table default.Table_4eb6a42428994acda28b9ba1bf7d5256 [id=162b3018a1a24d3391f815bd2efc8e94])
I20260926 01:56:35.782289 11251 tablet_loader.cc:96] loaded metadata for tablet cdd0f4921da4436aa259486227a180d5 (table default.Table_75d76cb423f64e279ac8c55a99a7b026 [id=a01eae184ad348b6a144e680e1974b29])
I20260926 01:56:35.793979 11251 tablet_loader.cc:96] loaded metadata for tablet d0bb5090e4874ae188b6d1ffe8649b4e (table default.Table_aeacfd2450264948a9933ded9cfcabd1 [id=5e6c5a34dd5249689909daf3e142a9b1])
I20260926 01:56:35.796257 11251 tablet_loader.cc:96] loaded metadata for tablet d0f4e81776f942ff96a9c1f274996c6a (table default.Table_4200bf9898c3412c9c196dda272aef75 [id=d6e69f9147414ae892b5766af1a21c96])
W20260926 01:56:35.838522 9155 consensus_peers.cc:597] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcb000f9234d4382ad019f63015b92e7. This is attempt 11: this message will repeat every 5th retry.
W20260926 01:56:35.847680 9910 leader_election.cc:336] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589): Timed out: RequestConsensusVote RPC to 127.8.76.190:44589 timed out after 0.807s (SENT)
I20260926 01:56:35.848168 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
W20260926 01:56:35.804453 9006 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 71150.1us
I20260926 01:56:35.863286 11296 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260926 01:56:35.866147 11251 tablet_loader.cc:96] loaded metadata for tablet d7e2b22ca59745879270e8dc20f2a4fa (table default.Table_4bbbc18bf03744a59245860ed3c2139e [id=a3195b6b2e634cab8b52bca361a17d2d])
I20260926 01:56:35.923996 11251 tablet_loader.cc:96] loaded metadata for tablet dbf222f9e14d4332b679f7ee94af1e0e (table default.Table_a2babbbea5d2444d8747712d56064903 [id=35630ee1a31d45eca3e0cf7ef7b995b6])
I20260926 01:56:35.937629 11251 tablet_loader.cc:96] loaded metadata for tablet dd22a450f5914523871a088b24795e9f (table default.Table_7f9e5bcaf9854a9b8de557a3b1a560fa [id=ded158519bae4b09b235e7e6ba605f90])
W20260926 01:56:36.024535 8843 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.131:43045, user_credentials={real_user=slave}} blocked reactor thread for 48436.6us
W20260926 01:56:36.054332 8948 debug-util.cc:398] Leaking SignalData structure 0x7b080006b680 after lost signal to thread 8801
W20260926 01:56:36.055791 8948 debug-util.cc:398] Leaking SignalData structure 0x7b080018ac80 after lost signal to thread 8951
I20260926 01:56:36.128306 11251 tablet_loader.cc:96] loaded metadata for tablet e140cf11b8bb49f9a94c0158a2afca4d (table default.Table_bbbb170151dd4078915478e24d1c1ee7 [id=9dc511f0c6e345b38e434da46a480c83])
I20260926 01:56:36.136053 11251 tablet_loader.cc:96] loaded metadata for tablet e18ddaa6f0e748cf930e29cbca3844b1 (table default.Table_ba018a2d02f94e7bb7c54c3496475dac [id=1f4236519087481abcfc98a06df0506b])
I20260926 01:56:36.137362 11251 tablet_loader.cc:96] loaded metadata for tablet e53c9eb8a6d648b2a9359f02b7f8764b (table default.Table_c085afec1ab24b588fa3ff21bc9f5d96 [id=fa498ebfcfaf4e4e8c9317e1738860d9])
I20260926 01:56:36.138516 11251 tablet_loader.cc:96] loaded metadata for tablet e55a31bb9b9d4862b6bcef71df0cdaea (table default.Table_a1fdc9d96e344c7bb928688dc2050a5c [id=4a025b4a52c248069b9f9954a015a932])
I20260926 01:56:36.156301 11251 tablet_loader.cc:96] loaded metadata for tablet e723112c1a6a4a6c89b6d17491d67c65 (table default.Table_da5237bb06194562a023680e328f087d [id=1c21bf55cfd54a3ca9fcb01d02491bfe])
I20260926 01:56:36.157641 11251 tablet_loader.cc:96] loaded metadata for tablet e7512e1d08e74ef6a2c699352347eb8d (table default.Table_d724813e115e41aaa5a65a7f4b262a3d [id=8965c9cf68bb4842a6bd0c6fa3576654])
I20260926 01:56:36.158846 11251 tablet_loader.cc:96] loaded metadata for tablet e76071c66b1d4635ac1a28110eb684fa (table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb])
I20260926 01:56:36.174890 11251 tablet_loader.cc:96] loaded metadata for tablet e85d17e9e48e4e54956e5c8460f7c9a8 (table default.Table_5f5bcfd270554939a7accb824ebd31d7 [id=3f5aa14692a1445cbf406a21d6b8541e])
I20260926 01:56:36.176338 11251 tablet_loader.cc:96] loaded metadata for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7])
I20260926 01:56:36.177397 11251 tablet_loader.cc:96] loaded metadata for tablet e8f5c3bdad494f34a3b72ab693599c55 (table default.Table_8992c2f5dc79410388cfe23af8c34f8b [id=2d8f343a5e944c06a878646559e51fdc])
I20260926 01:56:36.178575 11251 tablet_loader.cc:96] loaded metadata for tablet ea75f309937e4ee49b35558117847b05 (table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4])
I20260926 01:56:36.196615 11251 tablet_loader.cc:96] loaded metadata for tablet ec749c375d784786ba93020996708bb1 (table default.Table_6785898585374a828eb715d1bb3fef65 [id=8a4e463ee0c2406f930fb1c2cf1d97c6])
I20260926 01:56:36.199733 11251 tablet_loader.cc:96] loaded metadata for tablet edb7afa012944b3ba3c34bd994a95a83 (table default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134])
I20260926 01:56:36.200989 11251 tablet_loader.cc:96] loaded metadata for tablet ee418347ebb04927913810d539ef57ea (table default.Table_8b359d50e9ec420e921fdce8a36cb4f0 [id=25a2aac688fa4b99bc4674f751a7f743])
I20260926 01:56:36.204267 11251 tablet_loader.cc:96] loaded metadata for tablet f1f020590a8b428788e2bb516a11cc45 (table default.Table_d2e4b69665074e9898068bafd9ed0bb9 [id=2c5fb89a77a6445e8aef61c7eaf0603f])
I20260926 01:56:36.205535 11251 tablet_loader.cc:96] loaded metadata for tablet f2ee16b0c3e84936abba8248f194786a (table default.Table_f2f41e8f7ebd4d5fa3673a78cbca79d0 [id=182d678afe7743aba274cc33ba1a445c])
I20260926 01:56:36.207474 11251 tablet_loader.cc:96] loaded metadata for tablet f633f937b6c747f78efc6d3e167e83f0 (table default.Table_730735d338704c849d0e9a76c846e6cc [id=0e58ec4f07b143c3a4d1b9b6bd5735da])
I20260926 01:56:36.209326 11251 tablet_loader.cc:96] loaded metadata for tablet f74618f7c46744d2a08cb7b4e389c9f3 (table default.Table_dd0ec322addf465e90e656d847756e56 [id=fba6861106064ae5b5b05aaaa123dad4])
I20260926 01:56:36.210448 11251 tablet_loader.cc:96] loaded metadata for tablet f7dcb2358e434f36a831352105863807 (table default.Table_39851286a7944831b873cd10eb21d348 [id=85fcb50cb3b34f33b23595ee11d2493f])
I20260926 01:56:36.213726 11251 tablet_loader.cc:96] loaded metadata for tablet ffd47a7ddf854e0990882b460963ebb4 (table default.Table_b1a3c70f85824c9a9d3e73d3f7544f05 [id=21ac5392a7f64b5ea1052d99a885a2bf])
W20260926 01:56:36.240880 9155 consensus_peers.cc:597] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 -> Peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Couldn't send request to peer d20f26c9295641ecbf77d290ca206bb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16f49a5a10ea4f7a821d89f18e3af560. This is attempt 16: this message will repeat every 5th retry.
W20260926 01:56:36.369931 11251 catalog_manager.cc:1525] Time spent T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loading table and tablet metadata into memory: real 2.748s user 0.446s sys 0.070s
I20260926 01:56:36.370298 11251 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:56:36.370138 8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:36.371382 8542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 29) took 1330 ms (client timeout 807 ms). Trace:
W20260926 01:56:36.371762 8542 rpcz_store.cc:269] 0926 01:56:35.040510 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:35.040628 (+ 118us) service_pool.cc:224] Handling call
0926 01:56:36.371268 (+1330640us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:36.372896 8548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:36.373767 8548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 49) took 1933 ms (client timeout 508 ms). Trace:
W20260926 01:56:36.374037 8548 rpcz_store.cc:269] 0926 01:56:34.440051 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:34.440205 (+ 154us) service_pool.cc:224] Handling call
0926 01:56:36.373697 (+1933492us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:36.374926 8545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:36.375650 8545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44110 (request call id 50) took 1349 ms (client timeout 508 ms). Trace:
W20260926 01:56:36.375945 8545 rpcz_store.cc:269] 0926 01:56:35.026137 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:35.026419 (+ 282us) service_pool.cc:224] Handling call
0926 01:56:36.375589 (+1349170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:36.382378 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
W20260926 01:56:36.383261 8544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49078 (request call id 28) took 2161 ms (client timeout 807 ms). Trace:
W20260926 01:56:36.383554 8544 rpcz_store.cc:269] 0926 01:56:34.222163 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:34.222294 (+ 131us) service_pool.cc:224] Handling call
0926 01:56:36.383175 (+2160881us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:36.385196 8535 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:42048 (request call id 393) took 2674 ms. Trace:
I20260926 01:56:36.385519 8535 rpcz_store.cc:276] 0926 01:56:33.726569 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:56:33.726766 (+ 197us) service_pool.cc:224] Handling call
0926 01:56:36.384997 (+2658231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260926 01:56:36.364763 11298 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:36.404229 11298 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:36.419705 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:36.428042 9990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 12 candidate_status { last_received { term: 11 index: 362 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:36.432885 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:36.439407 11299 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260926 01:56:36.439888 11298 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:36.484789 11251 catalog_manager.cc:1308] Loaded cluster ID: e0d293d5ae834396ab84e80741be77e9
I20260926 01:56:36.485136 11251 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:56:36.566890 11251 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:56:36.601111 11251 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Loaded TSK: 0
I20260926 01:56:36.603863 11251 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:56:36.647274 8535 master_service.cc:960] ReplaceTablet: received request to replace tablet 030f4d683e404fbd8d2e6863e18f71c0 from {username='slave'} at 127.0.0.1:42048
I20260926 01:56:36.654098 8532 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8552d28bce1c484c964fe1268e370cc7" } modify_external_catalogs: true
I20260926 01:56:36.653888 8531 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42048:
name: "default.Table_8b014c6584dd427bbf5a612906ebb1b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:56:36.655009 8532 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42048:
table { table_name: "default.Table_8552d28bce1c484c964fe1268e370cc7" } modify_external_catalogs: true
I20260926 01:56:36.657015 8538 master_service.cc:960] ReplaceTablet: received request to replace tablet 880d77d42c394e6a9cf0b3768a915402 from {username='slave'} at 127.0.0.1:42048
W20260926 01:56:36.657538 8531 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b014c6584dd427bbf5a612906ebb1b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:56:36.806916 9187 tablet_service.cc:1558] Processing DeleteTablet for tablet bcb000f9234d4382ad019f63015b92e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:01 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:36.812784 11307 tablet_replica.cc:333] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:36.897500 11307 raft_consensus.cc:2243] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 2 LEADER]: Raft consensus shutting down.
I20260926 01:56:36.898690 11307 raft_consensus.cc:2272] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:36.901062 11307 ts_tablet_manager.cc:1916] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:36.924471 8954 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
W20260926 01:56:36.932637 11308 log.cc:927] Time spent T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5: Append to log took a long time: real 0.060s user 0.002s sys 0.000s
I20260926 01:56:36.953884 11307 ts_tablet_manager.cc:1929] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260926 01:56:36.959645 11307 log.cc:1199] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/bcb000f9234d4382ad019f63015b92e7
I20260926 01:56:36.960631 11307 ts_tablet_manager.cc:1950] T bcb000f9234d4382ad019f63015b92e7 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:36.967278 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet bcb000f9234d4382ad019f63015b92e7 (table default.Table_fe07ca51a9dd47b5a78a217875523300 [id=88aedd8765374c6395d82b3752c327ba]) successfully deleted
W20260926 01:56:36.988874 11309 log.cc:927] Time spent T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260926 01:56:37.075485 8532 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 5cbf646569e6497dac90e308f0681372
I20260926 01:56:37.079071 8874 tablet_service.cc:1558] Processing DeleteTablet for tablet 5cbf646569e6497dac90e308f0681372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:36 UTC) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:37.080911 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 5cbf646569e6497dac90e308f0681372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:36 UTC) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:37.081350 9187 tablet_service.cc:1558] Processing DeleteTablet for tablet 5cbf646569e6497dac90e308f0681372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 01:56:36 UTC) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:37.082280 11307 tablet_replica.cc:333] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:37.083250 11307 raft_consensus.cc:2243] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:37.083837 11307 raft_consensus.cc:2272] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:37.087435 11307 ts_tablet_manager.cc:1916] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:37.113258 11314 tablet_replica.cc:333] T 5cbf646569e6497dac90e308f0681372 P 49bd320bf73f4252804ec867ec077510: stopping tablet replica
I20260926 01:56:37.128800 11315 tablet_replica.cc:333] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:37.193740 8538 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 880d77d42c394e6a9cf0b3768a915402
I20260926 01:56:37.195417 8538 catalog_manager.cc:6934] ReplaceTablet: tablet 880d77d42c394e6a9cf0b3768a915402 deleted and replaced by tablet 7b71de5c4b504d068c34d8e4bbe59d9c
I20260926 01:56:37.198674 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 880d77d42c394e6a9cf0b3768a915402 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b71de5c4b504d068c34d8e4bbe59d9c) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:37.199577 8874 tablet_service.cc:1558] Processing DeleteTablet for tablet 880d77d42c394e6a9cf0b3768a915402 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b71de5c4b504d068c34d8e4bbe59d9c) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:37.203544 9187 tablet_service.cc:1558] Processing DeleteTablet for tablet 880d77d42c394e6a9cf0b3768a915402 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b71de5c4b504d068c34d8e4bbe59d9c) from {username='slave'} at 127.0.0.1:42408
W20260926 01:56:37.223004 8526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.189:34163, user_credentials={real_user=slave}} blocked reactor thread for 63351.6us
I20260926 01:56:37.225067 9267 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
I20260926 01:56:37.225690 8535 catalog_manager.cc:6133] T 00000000000000000000000000000000 P ca25983598bc4c0688340d8e7bc0f6ba: Sending DeleteTablet for 3 replicas of tablet 030f4d683e404fbd8d2e6863e18f71c0
I20260926 01:56:37.227339 8535 catalog_manager.cc:6934] ReplaceTablet: tablet 030f4d683e404fbd8d2e6863e18f71c0 deleted and replaced by tablet e4e04d9e4ee4452cb1a008cb73dafc31
I20260926 01:56:37.230412 9187 tablet_service.cc:1558] Processing DeleteTablet for tablet 030f4d683e404fbd8d2e6863e18f71c0 with delete_type TABLET_DATA_DELETED (replaced by tablet e4e04d9e4ee4452cb1a008cb73dafc31) from {username='slave'} at 127.0.0.1:42408
I20260926 01:56:37.231365 8874 tablet_service.cc:1558] Processing DeleteTablet for tablet 030f4d683e404fbd8d2e6863e18f71c0 with delete_type TABLET_DATA_DELETED (replaced by tablet e4e04d9e4ee4452cb1a008cb73dafc31) from {username='slave'} at 127.0.0.1:58226
I20260926 01:56:37.232086 9048 tablet_service.cc:1558] Processing DeleteTablet for tablet 030f4d683e404fbd8d2e6863e18f71c0 with delete_type TABLET_DATA_DELETED (replaced by tablet e4e04d9e4ee4452cb1a008cb73dafc31) from {username='slave'} at 127.0.0.1:51458
I20260926 01:56:37.233603 11320 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:37.234112 11320 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:37.251070 11315 raft_consensus.cc:2243] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:37.255957 11315 raft_consensus.cc:2272] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:37.258502 11315 ts_tablet_manager.cc:1916] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:37.264839 8498 master-stress-test.cc:581] Killed test thread 2/10
I20260926 01:56:37.265215 8498 master-stress-test.cc:579] Killing test thread 3/10
I20260926 01:56:37.265471 8498 master-stress-test.cc:581] Killed test thread 3/10
I20260926 01:56:37.265662 8498 master-stress-test.cc:579] Killing test thread 4/10
I20260926 01:56:37.265866 8498 master-stress-test.cc:581] Killed test thread 4/10
I20260926 01:56:37.266055 8498 master-stress-test.cc:579] Killing test thread 5/10
I20260926 01:56:37.266566 8498 master-stress-test.cc:581] Killed test thread 5/10
I20260926 01:56:37.266781 8498 master-stress-test.cc:579] Killing test thread 6/10
I20260926 01:56:37.267076 8498 master-stress-test.cc:581] Killed test thread 6/10
I20260926 01:56:37.267338 8498 master-stress-test.cc:579] Killing test thread 7/10
I20260926 01:56:37.267638 8498 master-stress-test.cc:581] Killed test thread 7/10
I20260926 01:56:37.267830 8498 master-stress-test.cc:579] Killing test thread 8/10
I20260926 01:56:37.268119 8498 master-stress-test.cc:581] Killed test thread 8/10
I20260926 01:56:37.268309 8498 master-stress-test.cc:579] Killing test thread 9/10
I20260926 01:56:37.268579 8498 master-stress-test.cc:581] Killed test thread 9/10
I20260926 01:56:37.268752 8498 master-stress-test.cc:579] Killing test thread 10/10
I20260926 01:56:37.269014 8498 master-stress-test.cc:581] Killed test thread 10/10
I20260926 01:56:37.269366 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 8791
I20260926 01:56:37.355322 11307 ts_tablet_manager.cc:1929] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:37.355966 11307 log.cc:1199] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/5cbf646569e6497dac90e308f0681372
I20260926 01:56:37.357931 11307 ts_tablet_manager.cc:1950] T 5cbf646569e6497dac90e308f0681372 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:37.368098 11320 leader_election.cc:290] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), 9ef98e01cb9a44408ebf207163f208b5 (127.8.76.189:34163)
I20260926 01:56:37.371835 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 12 candidate_status { last_received { term: 11 index: 363 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:37.372695 8624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f75ba46482de422ea23daf17aa100849" candidate_term: 12 candidate_status { last_received { term: 11 index: 363 } } ignore_live_leader: false dest_uuid: "9ef98e01cb9a44408ebf207163f208b5" is_pre_election: true
I20260926 01:56:37.454876 9910 leader_election.cc:304] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f75ba46482de422ea23daf17aa100849; no voters: 9ef98e01cb9a44408ebf207163f208b5, ca25983598bc4c0688340d8e7bc0f6ba
I20260926 01:56:37.475436 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 5cbf646569e6497dac90e308f0681372 (table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]) successfully deleted
I20260926 01:56:37.495827 11320 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f75ba46482de422ea23daf17aa100849 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260926 01:56:37.525988 11307 tablet_replica.cc:333] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:37.568692 11307 raft_consensus.cc:2243] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:37.569525 11307 raft_consensus.cc:2272] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:37.572327 11307 ts_tablet_manager.cc:1916] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:37.666316 11315 ts_tablet_manager.cc:1929] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:37.668272 11315 log.cc:1199] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/5cbf646569e6497dac90e308f0681372
I20260926 01:56:37.690412 9120 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
I20260926 01:56:37.703136 11315 ts_tablet_manager.cc:1950] T 5cbf646569e6497dac90e308f0681372 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:37.705168 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 5cbf646569e6497dac90e308f0681372 (table default.Table_8552d28bce1c484c964fe1268e370cc7 [id=859e1583e4a54bc1a5aeb5117f4bc61b]) successfully deleted
I20260926 01:56:37.706142 11315 tablet_replica.cc:333] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:37.707479 11315 raft_consensus.cc:2243] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:37.708425 11315 raft_consensus.cc:2272] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:37.727420 11328 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ca25983598bc4c0688340d8e7bc0f6ba)
I20260926 01:56:37.728295 11328 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 44589 } } peers { permanent_uuid: "9ef98e01cb9a44408ebf207163f208b5" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34163 } } peers { permanent_uuid: "f75ba46482de422ea23daf17aa100849" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 38143 } }
I20260926 01:56:37.731979 8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 12 candidate_status { last_received { term: 11 index: 366 } } ignore_live_leader: false dest_uuid: "ca25983598bc4c0688340d8e7bc0f6ba" is_pre_election: true
I20260926 01:56:37.742558 11307 ts_tablet_manager.cc:1929] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:37.743083 11307 log.cc:1199] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/880d77d42c394e6a9cf0b3768a915402
I20260926 01:56:37.743983 11307 ts_tablet_manager.cc:1950] T 880d77d42c394e6a9cf0b3768a915402 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:37.745893 11315 ts_tablet_manager.cc:1916] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:37.747265 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 880d77d42c394e6a9cf0b3768a915402 (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]) successfully deleted
I20260926 01:56:37.748904 11307 tablet_replica.cc:333] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: stopping tablet replica
I20260926 01:56:37.749001 9990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ef98e01cb9a44408ebf207163f208b5" candidate_term: 12 candidate_status { last_received { term: 11 index: 366 } } ignore_live_leader: false dest_uuid: "f75ba46482de422ea23daf17aa100849" is_pre_election: true
I20260926 01:56:37.749989 11307 raft_consensus.cc:2243] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:37.750593 11307 raft_consensus.cc:2272] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:37.751039 8599 leader_election.cc:304] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef98e01cb9a44408ebf207163f208b5; no voters: ca25983598bc4c0688340d8e7bc0f6ba, f75ba46482de422ea23daf17aa100849
I20260926 01:56:37.754048 11307 ts_tablet_manager.cc:1916] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:37.756176 11328 leader_election.cc:290] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ca25983598bc4c0688340d8e7bc0f6ba (127.8.76.190:44589), f75ba46482de422ea23daf17aa100849 (127.8.76.188:38143)
I20260926 01:56:37.756786 11328 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9ef98e01cb9a44408ebf207163f208b5 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260926 01:56:37.774530 11307 ts_tablet_manager.cc:1929] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:37.775807 11307 log.cc:1199] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/030f4d683e404fbd8d2e6863e18f71c0
I20260926 01:56:37.776665 11307 ts_tablet_manager.cc:1950] T 030f4d683e404fbd8d2e6863e18f71c0 P bc00feec3f564608851f0112e97ab532: Deleting consensus metadata
I20260926 01:56:37.779829 8527 catalog_manager.cc:5063] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045): tablet 030f4d683e404fbd8d2e6863e18f71c0 (table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]) successfully deleted
I20260926 01:56:37.860288 11315 ts_tablet_manager.cc:1929] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:37.860740 11315 log.cc:1199] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/880d77d42c394e6a9cf0b3768a915402
I20260926 01:56:37.861536 11315 ts_tablet_manager.cc:1950] T 880d77d42c394e6a9cf0b3768a915402 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:37.864636 11315 tablet_replica.cc:333] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: stopping tablet replica
I20260926 01:56:37.864641 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 880d77d42c394e6a9cf0b3768a915402 (table default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]) successfully deleted
I20260926 01:56:37.865568 11315 raft_consensus.cc:2243] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:56:37.866184 11315 raft_consensus.cc:2272] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:56:37.886968 11315 ts_tablet_manager.cc:1916] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:56:37.935627 11315 ts_tablet_manager.cc:1929] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:56:37.936074 11315 log.cc:1199] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/030f4d683e404fbd8d2e6863e18f71c0
I20260926 01:56:37.936859 11315 ts_tablet_manager.cc:1950] T 030f4d683e404fbd8d2e6863e18f71c0 P d20f26c9295641ecbf77d290ca206bb2: Deleting consensus metadata
I20260926 01:56:37.955333 8527 catalog_manager.cc:5063] TS d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): tablet 030f4d683e404fbd8d2e6863e18f71c0 (table default.Table_bd491a7ccf524a75a6f05225b003cdc0 [id=11824f3450264215a9a6e0b76df29d72]) successfully deleted
I20260926 01:56:37.988255 9187 tablet_service.cc:1511] Processing CreateTablet for tablet 3a7f7eaf2d444886894b3b39902be4c9 (DEFAULT_TABLE table=default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:37.988324 9048 tablet_service.cc:1511] Processing CreateTablet for tablet 3a7f7eaf2d444886894b3b39902be4c9 (DEFAULT_TABLE table=default.Table_bcebd87d7c6f4822bb6a50fd4e9999d4 [id=f5f1e7c0e86147e795e4c7d72387d79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:37.989751 9048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7f7eaf2d444886894b3b39902be4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:37.995638 9051 tablet_service.cc:1511] Processing CreateTablet for tablet 1b84d9bc9c6f41bd80482c129e569648 (DEFAULT_TABLE table=default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:37.996793 9051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b84d9bc9c6f41bd80482c129e569648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:37.999372 9190 tablet_service.cc:1511] Processing CreateTablet for tablet 1b84d9bc9c6f41bd80482c129e569648 (DEFAULT_TABLE table=default.Table_82723302583843e2b50ce4a3d0e4adcd [id=dc12b3bed7754545a0723fd24d96247b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.000478 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7f7eaf2d444886894b3b39902be4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.001117 9045 tablet_service.cc:1511] Processing CreateTablet for tablet af0840d6761a4521b0b6136b80eb6a6c (DEFAULT_TABLE table=default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.002205 9045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0840d6761a4521b0b6136b80eb6a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.007592 9192 tablet_service.cc:1511] Processing CreateTablet for tablet af0840d6761a4521b0b6136b80eb6a6c (DEFAULT_TABLE table=default.Table_4fdfd0dfce964d90b625381eabd6eac5 [id=e64cf03d172a48b88cf2dc1a1ae1833f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.010054 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0840d6761a4521b0b6136b80eb6a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.019588 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b84d9bc9c6f41bd80482c129e569648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.027415 9050 tablet_service.cc:1511] Processing CreateTablet for tablet edb7afa012944b3ba3c34bd994a95a83 (DEFAULT_TABLE table=default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.028566 9050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb7afa012944b3ba3c34bd994a95a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.028702 9196 tablet_service.cc:1511] Processing CreateTablet for tablet edb7afa012944b3ba3c34bd994a95a83 (DEFAULT_TABLE table=default.Table_72cdae2a885f4338977c6aed96a2cdcb [id=ba54ad81457b42fb80a3a849d5de9134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.029852 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb7afa012944b3ba3c34bd994a95a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.048418 9052 tablet_service.cc:1511] Processing CreateTablet for tablet 9d973538ff534f05b301cb619475d4f3 (DEFAULT_TABLE table=default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.049604 9052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d973538ff534f05b301cb619475d4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.061818 9184 tablet_service.cc:1511] Processing CreateTablet for tablet 9d973538ff534f05b301cb619475d4f3 (DEFAULT_TABLE table=default.Table_d908977ed46746238f564a8c31a4c83e [id=a2225b255da742799264a8606db3c46a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.063088 9184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d973538ff534f05b301cb619475d4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.072264 11334 tablet_bootstrap.cc:492] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
I20260926 01:56:38.087646 9045 tablet_service.cc:1511] Processing CreateTablet for tablet 7b71de5c4b504d068c34d8e4bbe59d9c (DEFAULT_TABLE table=default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.087716 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 7b71de5c4b504d068c34d8e4bbe59d9c (DEFAULT_TABLE table=default.Table_a6bc2356bd5e45899d0360430fe873e6 [id=6c29a0d299b34f55aa0a785fc93399b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.088793 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b71de5c4b504d068c34d8e4bbe59d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.096469 9045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b71de5c4b504d068c34d8e4bbe59d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.101337 9048 tablet_service.cc:1511] Processing CreateTablet for tablet bce97ff01117487a8fee85b5349679a2 (DEFAULT_TABLE table=default.Table_619cd7cf639645f6a3039752fc75e5f6 [id=beae83b06ad14413b3ecff25b28af923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.102509 9048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce97ff01117487a8fee85b5349679a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.104943 11334 tablet_bootstrap.cc:654] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2: Neither blocks nor log segments found. Creating new log.
W20260926 01:56:38.107324 9003 connection.cc:573] client connection to 127.8.76.129:43257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.107810 9003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.107887 8524 connection.cc:573] client connection to 127.8.76.129:43257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.108314 8524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.124132 9152 connection.cc:573] client connection to 127.8.76.129:43257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.124671 9152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.131310 9003 connection.cc:573] server connection from 127.8.76.129:48977 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 01:56:38.139671 11334 tablet_bootstrap.cc:492] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2: No bootstrap required, opened a new log
I20260926 01:56:38.140479 11334 ts_tablet_manager.cc:1403] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.003s
W20260926 01:56:38.145911 11338 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:56:38.119016 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.8.76.129:43257 (local address 127.8.76.130:45709)
0926 01:56:38.128922 (+ 9906us) negotiation.cc:107] Waiting for socket to connect
0926 01:56:38.128996 (+ 74us) client_negotiation.cc:175] Beginning negotiation
0926 01:56:38.129415 (+ 419us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0926 01:56:38.144297 (+ 14882us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":9121,"thread_start_us":10457,"threads_started":1}
W20260926 01:56:38.149324 9003 consensus_peers.cc:597] T 215a061440854c56b00df9fda9192598 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.151587 9003 consensus_peers.cc:597] T 3f1d71caf78f4c75b144af14dce19d70 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.152139 9003 consensus_peers.cc:597] T 8711788a85f94c599224123f6b8cc814 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.153434 9003 consensus_peers.cc:597] T 6d101ad03f8e4d7f8bf70ea993c4f58b P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.143779 11334 raft_consensus.cc:359] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
W20260926 01:56:38.154523 9003 consensus_peers.cc:597] T b612e8955d5f42dd8658320031fe36ca P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.155334 9003 consensus_peers.cc:597] T c448caeeedfb4fc88eee88e40eebfeeb P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.156301 9003 consensus_peers.cc:597] T b4c64a7619c4452f94b8d8b4f8bbbcf8 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.156730 9003 consensus_peers.cc:597] T a6e8059b94564c42894c6c8a3d695950 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.157155 9003 consensus_peers.cc:597] T 6752df6a58914df5b8718dd0439aa885 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.157555 9003 consensus_peers.cc:597] T 5eef6a48e87e4c61a23058942a82bccc P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.157956 9003 consensus_peers.cc:597] T 6f7419abcc2c482e97df70f03e047450 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.158363 9003 consensus_peers.cc:597] T 3c7151ffe5d94a0599bbe3e6c8999371 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.158818 9003 consensus_peers.cc:597] T e7512e1d08e74ef6a2c699352347eb8d P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.154318 11334 raft_consensus.cc:385] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:38.159288 11334 raft_consensus.cc:740] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20f26c9295641ecbf77d290ca206bb2, State: Initialized, Role: FOLLOWER
I20260926 01:56:38.160410 11334 consensus_queue.cc:260] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:38.167007 8531 catalog_manager.cc:5838] T 6ab082379974486bbea9f87f5aa64e10 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.168478 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 6ab082379974486bbea9f87f5aa64e10 (table default.Table_370cc280a4a6451cb1dae27ab9fce101 [id=92cd6d238855439da8e5715362e7d2da]). Expected version 1 got 0
I20260926 01:56:38.169603 8531 catalog_manager.cc:5838] T b85e2acbec2e485cb1b9728bd5763b0e P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.173036 11334 ts_tablet_manager.cc:1434] T 9d973538ff534f05b301cb619475d4f3 P d20f26c9295641ecbf77d290ca206bb2: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260926 01:56:38.174011 11334 tablet_bootstrap.cc:492] T 3a7f7eaf2d444886894b3b39902be4c9 P d20f26c9295641ecbf77d290ca206bb2: Bootstrap starting.
W20260926 01:56:38.174469 9003 consensus_peers.cc:597] T a09351c3db384b4383ca5ff0eaca982c P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.177335 9003 consensus_peers.cc:597] T 05042bcc0a6e426d9e3e8b25d8d55540 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.177788 9003 consensus_peers.cc:597] T 0c6cccff3de64f3384ce58bb84c43487 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.178196 9003 consensus_peers.cc:597] T accc62f46ce24a4d97743dca05e65f51 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.179400 9003 consensus_peers.cc:597] T 030c41a9718d47ddaa33382b20a60dab P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.179817 9003 consensus_peers.cc:597] T 5f300124bc3a4177a606b116bfef3a65 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.180197 9003 consensus_peers.cc:597] T 0558d37a3b5f4bcea43a8da7b99ded09 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.180927 9003 consensus_peers.cc:597] T b778c95327fc477ca210cb3c983665c1 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.181344 9003 consensus_peers.cc:597] T 3c26f557ac274973824e6f9a4532de74 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.181737 9003 consensus_peers.cc:597] T 272bb87fe22c4720bcb3f0e07482dd8b P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.182147 9003 consensus_peers.cc:597] T f74618f7c46744d2a08cb7b4e389c9f3 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.182543 9003 consensus_peers.cc:597] T e55a31bb9b9d4862b6bcef71df0cdaea P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.188102 8531 catalog_manager.cc:5838] T 5f300124bc3a4177a606b116bfef3a65 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
W20260926 01:56:38.199096 9003 consensus_peers.cc:597] T 1c8761a6c3f7435193a91bfc4fe7fa86 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.192553 8524 catalog_manager.cc:4790] TS 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cbf646569e6497dac90e308f0681372: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111)
W20260926 01:56:38.199575 9003 consensus_peers.cc:597] T 89fb965257de4db6b043b4a6dbe22709 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.199923 9003 consensus_peers.cc:597] T e8e0e6ede1ce4487b08bfd5743e3656a P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.200250 9003 consensus_peers.cc:597] T 681900e0fac74cdabaea1d8a7fb4a189 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.200567 9003 consensus_peers.cc:597] T 7e57300d2bcf4e0496ed13a4e7d967f3 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.200913 9003 consensus_peers.cc:597] T f2ee16b0c3e84936abba8248f194786a P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.201259 9003 consensus_peers.cc:597] T 8aae2a23d3fe478e928757db4b30f5ca P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.201598 9003 consensus_peers.cc:597] T e53c9eb8a6d648b2a9359f02b7f8764b P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.201972 9003 consensus_peers.cc:597] T 4411120140f3410c9f4e33e26da20d98 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.202315 9003 consensus_peers.cc:597] T 1c5c66728bb54ec6beb4f0a2c25d9105 P d20f26c9295641ecbf77d290ca206bb2 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.209004 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 8962
I20260926 01:56:38.199400 8531 catalog_manager.cc:5838] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 2, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 2 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } }
I20260926 01:56:38.219436 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 70d2447335b34731a4cb2cd0f3b4bf13 (table default.Table_f8b2b30433ac4b2c95082c4b0199bce9 [id=311093f5896c4393b682c14217374e2e]). Expected version 1 got 0
I20260926 01:56:38.221637 9184 tablet_service.cc:1511] Processing CreateTablet for tablet bce97ff01117487a8fee85b5349679a2 (DEFAULT_TABLE table=default.Table_619cd7cf639645f6a3039752fc75e5f6 [id=beae83b06ad14413b3ecff25b28af923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.223016 9184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce97ff01117487a8fee85b5349679a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.231650 9192 tablet_service.cc:1511] Processing CreateTablet for tablet 1279a644e6604fb0bc20f53cd23dd8be (DEFAULT_TABLE table=default.Table_3951d686ab954ed784d5921048abb4a5 [id=c241dc156c114a4684de8b3280984f1a]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:56:38.232368 9152 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.76.129:43257, user_credentials={real_user=slave}} blocked reactor thread for 107757us
I20260926 01:56:38.236948 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1279a644e6604fb0bc20f53cd23dd8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.240583 11340 tablet_bootstrap.cc:492] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:38.250535 8531 catalog_manager.cc:5838] T cdd0f4921da4436aa259486227a180d5 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.258622 9195 tablet_service.cc:1511] Processing CreateTablet for tablet 0b2a99b3bfd64925a82236cec1d9e71f (DEFAULT_TABLE table=default.Table_05db3de2a0574423980de7d7b7098324 [id=5226bf119e6d4ffb839aea00e909e760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.259857 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2a99b3bfd64925a82236cec1d9e71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.267800 8531 catalog_manager.cc:5838] T b25107dd5c104c83a2a21847bb2fc24f P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.278744 8531 catalog_manager.cc:5838] T ea75f309937e4ee49b35558117847b05 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
W20260926 01:56:38.279556 9152 consensus_peers.cc:597] T 28eaf7381f364a2682dd07f2bd90ede3 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.279884 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet ea75f309937e4ee49b35558117847b05 (table default.Table_8004b577262548dbbc92c297b99e01b2 [id=2f95b129dc704c41b6a54ed2ef954eb4]). Expected version 1 got 0
W20260926 01:56:38.280009 9152 consensus_peers.cc:597] T 70d2447335b34731a4cb2cd0f3b4bf13 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.280349 9152 consensus_peers.cc:597] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.280691 9152 consensus_peers.cc:597] T 04e79a2e52fb40619b2bc99396abda5b P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.281028 9152 consensus_peers.cc:597] T 8dfff96cc29145e7a72213bb5ecca1ae P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.281373 9152 consensus_peers.cc:597] T 16f49a5a10ea4f7a821d89f18e3af560 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.284013 9195 tablet_service.cc:1511] Processing CreateTablet for tablet a73602570425443d97d28fcebe3cc6fc (DEFAULT_TABLE table=default.Table_818d89431f354c43ac77ba53fcb9cd8e [id=50846b080147404f91b349751918cca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.286437 11340 tablet_bootstrap.cc:654] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:38.292876 8531 catalog_manager.cc:5838] T 1c8761a6c3f7435193a91bfc4fe7fa86 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.303089 8531 catalog_manager.cc:5838] T e55a31bb9b9d4862b6bcef71df0cdaea P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.288254 9195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73602570425443d97d28fcebe3cc6fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:38.292670 9152 consensus_peers.cc:597] T c9f54f648a55404fb6622b35d84368ee P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.305898 9152 consensus_peers.cc:597] T 229f4cdd841949e4a12e8555cdfe42cc P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.306325 9192 tablet_service.cc:1511] Processing CreateTablet for tablet 6cbfa4fa8ab146108cd2c62a0de90b40 (DEFAULT_TABLE table=default.Table_f751bf0f1a19437aade46f4760f6f69b [id=d8ddf10a94d3432b9fd6eac6f6c27e95]), partition=RANGE (key) PARTITION UNBOUNDED
W20260926 01:56:38.306324 9152 consensus_peers.cc:597] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.307446 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cbfa4fa8ab146108cd2c62a0de90b40. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:56:38.316212 9152 consensus_peers.cc:597] T ee418347ebb04927913810d539ef57ea P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.316706 9152 consensus_peers.cc:597] T 562604137d484e83a51de2f778d58fca P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260926 01:56:38.317076 9152 consensus_peers.cc:597] T 661148692f9e4127af9abf5cc6cc0f54 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.320757 9196 tablet_service.cc:1511] Processing CreateTablet for tablet 572b97ad3aef4dce8cb3bcb521e90e48 (DEFAULT_TABLE table=default.Table_c5ca9108f53e46bbb3d6acf830b83ce1 [id=30c67008e5de4b3da4aac5964c64c346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.324290 8531 catalog_manager.cc:5838] T 05042bcc0a6e426d9e3e8b25d8d55540 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.328066 9190 tablet_service.cc:1511] Processing CreateTablet for tablet 0ae1f842516445588886ed7e7149c02a (DEFAULT_TABLE table=default.Table_7d32da88077d4060913cc8ae86df83f6 [id=d2357d68ebb749358cb06a52c588b99f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.329648 9196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572b97ad3aef4dce8cb3bcb521e90e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.335152 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ae1f842516445588886ed7e7149c02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.339917 9267 heartbeater.cc:499] Master 127.8.76.190:44589 was elected leader, sending a full tablet report...
I20260926 01:56:38.346180 9192 tablet_service.cc:1511] Processing CreateTablet for tablet 225bcc7bd4884188bba09b331b5da8e2 (DEFAULT_TABLE table=default.Table_0a43386150b14b53ad954d8f63768297 [id=27765f1075e54d509228a77011d6c8e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.347406 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225bcc7bd4884188bba09b331b5da8e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.352093 9184 tablet_service.cc:1511] Processing CreateTablet for tablet ffd47a7ddf854e0990882b460963ebb4 (DEFAULT_TABLE table=default.Table_b1a3c70f85824c9a9d3e73d3f7544f05 [id=21ac5392a7f64b5ea1052d99a885a2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.353543 9184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd47a7ddf854e0990882b460963ebb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.363497 11340 tablet_bootstrap.cc:492] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:38.364138 11340 ts_tablet_manager.cc:1403] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.124s user 0.005s sys 0.012s
I20260926 01:56:38.367316 11340 raft_consensus.cc:359] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:38.368183 11340 raft_consensus.cc:385] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:38.369232 11340 raft_consensus.cc:740] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:38.369416 8531 catalog_manager.cc:5838] T 4623456312284a698ae69d4054909647 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.370675 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 4623456312284a698ae69d4054909647 (table default.Table_ed104ff312a24200ac22b165f4981c96 [id=2103f22c921d4213827d63854da752aa]). Expected version 1 got 0
I20260926 01:56:38.371568 9187 tablet_service.cc:1511] Processing CreateTablet for tablet 14926a101b404f6b8d5a35b2aadc6c39 (DEFAULT_TABLE table=default.Table_bf92ccd2dbe1453d943f371ffec0badb [id=1cd337ec9ec94a47b24e774d0fff3321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.372673 9187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14926a101b404f6b8d5a35b2aadc6c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.381145 11340 consensus_queue.cc:260] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:38.393195 9197 tablet_service.cc:1511] Processing CreateTablet for tablet 38a52e650df5449c86924387a0535d39 (DEFAULT_TABLE table=default.Table_e04a4278262e4d7f8d7a94ddf8666bbe [id=7879fa789a5143a6a1bb95768cf1bf50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.399296 11340 ts_tablet_manager.cc:1434] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260926 01:56:38.371963 8531 catalog_manager.cc:5838] T e18ddaa6f0e748cf930e29cbca3844b1 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.405098 8531 catalog_manager.cc:5838] T 681900e0fac74cdabaea1d8a7fb4a189 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 2, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 2 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } }
I20260926 01:56:38.406205 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 681900e0fac74cdabaea1d8a7fb4a189 (table default.Table_aa85c048a6eb4c18a82c25546cb20413 [id=d9d802b9f0f34b9399be344e5897a1f2]). Expected version 1 got 0
I20260926 01:56:38.406890 8531 catalog_manager.cc:5838] T 35ffd8b0a9a44ae18a2d0e383a5b7724 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.407845 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 35ffd8b0a9a44ae18a2d0e383a5b7724 (table default.Table_aa6bb98036044d52970008af2616d33c [id=7eccc1870b21429ea89fa66914b317dd]). Expected version 1 got 0
I20260926 01:56:38.408730 8531 catalog_manager.cc:5838] T 812fe778734d4210b744ed8e45d412f1 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
W20260926 01:56:38.412354 9152 consensus_peers.cc:597] T f1f020590a8b428788e2bb516a11cc45 P bc00feec3f564608851f0112e97ab532 -> Peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Couldn't send request to peer 49bd320bf73f4252804ec867ec077510. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:56:38.410025 8531 catalog_manager.cc:5838] T 0fdb7a7328c54031bd3afac1636d0f75 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.424911 8531 catalog_manager.cc:5838] T 1c5c66728bb54ec6beb4f0a2c25d9105 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.426296 8531 catalog_manager.cc:5838] T 6f7419abcc2c482e97df70f03e047450 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.429898 9198 tablet_service.cc:1511] Processing CreateTablet for tablet e723112c1a6a4a6c89b6d17491d67c65 (DEFAULT_TABLE table=default.Table_da5237bb06194562a023680e328f087d [id=1c21bf55cfd54a3ca9fcb01d02491bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.438279 9197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a52e650df5449c86924387a0535d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.439249 11340 tablet_bootstrap.cc:492] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:38.441071 8531 catalog_manager.cc:5838] T 1f10cb68586b4a73abb86641b90c12f7 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } }
I20260926 01:56:38.442385 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 1f10cb68586b4a73abb86641b90c12f7 (table default.Table_f76e778fb54a460b89d85239e3fe34dd [id=8e41fb91eafb42aa9cf99435ebb8371f]). Expected version 1 got 0
I20260926 01:56:38.443275 8531 catalog_manager.cc:5838] T dbf222f9e14d4332b679f7ee94af1e0e P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.444737 8531 catalog_manager.cc:5838] T accc62f46ce24a4d97743dca05e65f51 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.438306 9190 tablet_service.cc:1511] Processing CreateTablet for tablet f633f937b6c747f78efc6d3e167e83f0 (DEFAULT_TABLE table=default.Table_730735d338704c849d0e9a76c846e6cc [id=0e58ec4f07b143c3a4d1b9b6bd5735da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:56:38.446748 9198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e723112c1a6a4a6c89b6d17491d67c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.446094 8531 catalog_manager.cc:5838] T 5da0c41e2e19409d938f8f8ec29cb91a P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc00feec3f564608851f0112e97ab532 (127.8.76.131). New cstate: current_term: 1 leader_uuid: "bc00feec3f564608851f0112e97ab532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } health_report { overall_health: HEALTHY } } }
I20260926 01:56:38.447654 9190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f633f937b6c747f78efc6d3e167e83f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:38.447629 8531 catalog_manager.cc:5838] T b778c95327fc477ca210cb3c983665c1 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.451311 11340 tablet_bootstrap.cc:654] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532: Neither blocks nor log segments found. Creating new log.
I20260926 01:56:38.450739 8531 catalog_manager.cc:5838] T 3c7151ffe5d94a0599bbe3e6c8999371 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.452083 8531 catalog_manager.cc:5838] T e76071c66b1d4635ac1a28110eb684fa P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.453074 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet e76071c66b1d4635ac1a28110eb684fa (table default.Table_5f2783bf7cb9432f86a847b22c398ea4 [id=2fa9e93da5704e3b9970e16f5afe93bb]). Expected version 1 got 0
I20260926 01:56:38.468008 8531 catalog_manager.cc:5838] T e8e0e6ede1ce4487b08bfd5743e3656a P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.469409 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet e8e0e6ede1ce4487b08bfd5743e3656a (table default.Table_c123991aeaf149f4a35bd18938938414 [id=6a3cae01c8664f3d8d4b89e8a91665f7]). Expected version 4 got 0
I20260926 01:56:38.470160 8531 catalog_manager.cc:5838] T 89fb965257de4db6b043b4a6dbe22709 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.471827 8531 catalog_manager.cc:5838] T 215a061440854c56b00df9fda9192598 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.473425 8531 catalog_manager.cc:5838] T 5eef6a48e87e4c61a23058942a82bccc P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.474545 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 5eef6a48e87e4c61a23058942a82bccc (table default.Table_4a4b59ee98dd4139b83d181413443621 [id=37400e3a991842198609512f0a9aeead]). Expected version 1 got 0
I20260926 01:56:38.475224 8531 catalog_manager.cc:5838] T f74618f7c46744d2a08cb7b4e389c9f3 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.476559 8531 catalog_manager.cc:5838] T e85d17e9e48e4e54956e5c8460f7c9a8 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.477838 8531 catalog_manager.cc:5838] T 8711788a85f94c599224123f6b8cc814 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.479223 8531 catalog_manager.cc:5838] T 094b7485380b46fe8e1fef23ac50b830 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.480576 8531 catalog_manager.cc:5838] T 8aae2a23d3fe478e928757db4b30f5ca P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 2, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 2 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.481827 8531 catalog_manager.cc:5838] T e140cf11b8bb49f9a94c0158a2afca4d P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.483132 8531 catalog_manager.cc:5838] T 7e57300d2bcf4e0496ed13a4e7d967f3 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.484663 8531 catalog_manager.cc:5838] T ec749c375d784786ba93020996708bb1 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.485948 8531 catalog_manager.cc:5838] T 6629d125861f4e27b61c10d585a35c10 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.489657 11301 raft_consensus.cc:493] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:56:38.490154 11301 raft_consensus.cc:515] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:38.492300 11301 leader_election.cc:290] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257), d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395)
I20260926 01:56:38.488662 8531 catalog_manager.cc:5838] T e7512e1d08e74ef6a2c699352347eb8d P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.494208 8531 catalog_manager.cc:5838] T d7e2b22ca59745879270e8dc20f2a4fa P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.495610 8531 catalog_manager.cc:5838] T 504cca4587d64bc4964b298ba1a74fe9 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.496904 8531 catalog_manager.cc:5838] T 2c0cd3e30f084966b27921e658cacf58 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.498150 8531 catalog_manager.cc:5838] T dd22a450f5914523871a088b24795e9f P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.499637 11340 tablet_bootstrap.cc:492] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532: No bootstrap required, opened a new log
I20260926 01:56:38.499457 8531 catalog_manager.cc:5838] T 3f1d71caf78f4c75b144af14dce19d70 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.500141 11340 ts_tablet_manager.cc:1403] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.000s
I20260926 01:56:38.500816 8531 catalog_manager.cc:5838] T d0f4e81776f942ff96a9c1f274996c6a P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 2, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 2 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.502066 8531 catalog_manager.cc:5838] T 4efbc1e1ef35420db51742d99d5a32bf P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.503445 8531 catalog_manager.cc:5838] T c450d101aa2b47c0bac3d4964c7bd6bd P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.504726 8531 catalog_manager.cc:5838] T a6e8059b94564c42894c6c8a3d695950 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.505749 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet a6e8059b94564c42894c6c8a3d695950 (table default.Table_4742d4d5469a49b3be1b4c39d6ef9b4c [id=d680dfae1d21415382be575ea6696a44]). Expected version 1 got 0
I20260926 01:56:38.506564 8531 catalog_manager.cc:5838] T 3c26f557ac274973824e6f9a4532de74 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
W20260926 01:56:38.507769 9152 leader_election.cc:336] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 49bd320bf73f4252804ec867ec077510 (127.8.76.129:43257): Network error: Client connection negotiation failed: client connection to 127.8.76.129:43257: connect: Connection refused (error 111)
I20260926 01:56:38.508096 11340 raft_consensus.cc:359] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:38.508121 8531 catalog_manager.cc:5838] T 0558d37a3b5f4bcea43a8da7b99ded09 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.509222 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 0558d37a3b5f4bcea43a8da7b99ded09 (table default.Table_7606d0ec00eb48549e0e381fdc608430 [id=05847c8c0e744754b5456a6773f5e41f]). Expected version 1 got 0
I20260926 01:56:38.510151 8531 catalog_manager.cc:5838] T 6752df6a58914df5b8718dd0439aa885 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130). New cstate: current_term: 1 leader_uuid: "d20f26c9295641ecbf77d290ca206bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } }
I20260926 01:56:38.511228 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 6752df6a58914df5b8718dd0439aa885 (table default.Table_fc05e029f86e4b1fa2b981970ef38da8 [id=0c2fc4ba7a0846b0a30cfc1ca6211e63]). Expected version 1 got 0
I20260926 01:56:38.512012 8531 catalog_manager.cc:5838] T 08b1d98089fd499a8e1ee217b561a801 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.512874 11340 raft_consensus.cc:385] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:56:38.513187 11340 raft_consensus.cc:740] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc00feec3f564608851f0112e97ab532, State: Initialized, Role: FOLLOWER
I20260926 01:56:38.513435 8531 catalog_manager.cc:5838] T 9db85241141f4a54a2b7dce15412a6f9 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.514918 8531 catalog_manager.cc:5838] T 1c1d70856f5c4ac68a5fcc8b5e1ef89c P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.515630 11340 consensus_queue.cc:260] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } }
I20260926 01:56:38.517987 11340 ts_tablet_manager.cc:1434] T 3a7f7eaf2d444886894b3b39902be4c9 P bc00feec3f564608851f0112e97ab532: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260926 01:56:38.516184 8531 catalog_manager.cc:5838] T 2a093fc10b604aa491b6e0f9596a589d P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.520781 8531 catalog_manager.cc:5838] T 91964074a37441db8b09f78720159586 P bc00feec3f564608851f0112e97ab532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bd320bf73f4252804ec867ec077510 (127.8.76.129). New cstate: current_term: 1 leader_uuid: "49bd320bf73f4252804ec867ec077510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc00feec3f564608851f0112e97ab532" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 43045 } } peers { permanent_uuid: "49bd320bf73f4252804ec867ec077510" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 43257 } } peers { permanent_uuid: "d20f26c9295641ecbf77d290ca206bb2" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 41395 } } }
I20260926 01:56:38.531538 11340 tablet_bootstrap.cc:492] T 1b84d9bc9c6f41bd80482c129e569648 P bc00feec3f564608851f0112e97ab532: Bootstrap starting.
I20260926 01:56:38.532128 8531 catalog_manager.cc:5938] TS bc00feec3f564608851f0112e97ab532 (127.8.76.131:43045) does not have the latest schema for tablet 91964074a37441db8b09f78720159586 (table default.Table_a21182bd8e5043a99c406b2f1ed0f766 [id=c2695ea31dc14f2181a9b475de86e5e9]). Expected version 1 got 0
W20260926 01:56:38.543694 9925 connection.cc:573] server connection from 127.8.76.130:42977 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.543694 9152 connection.cc:573] server connection from 127.8.76.130:53989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.550691 9155 connection.cc:573] client connection to 127.8.76.130:41395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.551138 8601 connection.cc:573] server connection from 127.8.76.130:45011 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.551791 8527 connection.cc:573] client connection to 127.8.76.130:41395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:56:38.554579 9155 leader_election.cc:336] T 7b71de5c4b504d068c34d8e4bbe59d9c P bc00feec3f564608851f0112e97ab532 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d20f26c9295641ecbf77d290ca206bb2 (127.8.76.130:41395): Network error: Client connection negotiation failed: client connection to 127.8.76.130:41395: connect: Connection refused (error 111)
I20260926 01:56:38.564428 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 9128
I20260926 01:56:38.757434 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 8507
I20260926 01:56:38.822613 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 8577
I20260926 01:56:38.861508 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 9349
2026-09-26T01:56:38Z chronyd exiting
I20260926 01:56:38.926393 8498 master-stress-test.cc:587] Tables created: 149
I20260926 01:56:38.926726 8498 master-stress-test.cc:588] Tables altered: 65
I20260926 01:56:38.926903 8498 master-stress-test.cc:589] Tables deleted: 53
I20260926 01:56:38.927057 8498 master-stress-test.cc:590] Tablets replaced: 33
I20260926 01:56:38.927194 8498 master-stress-test.cc:591] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (75463 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
2026-09-26T01:56:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-26T01:56:39Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskiRTH_v/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskiRTH_v/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/security.properties
2026-09-26 01:56:46: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskiRTH_v/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskiRTH_v/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260926 01:56:56.550858 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.190:41989
--webserver_interface=127.8.76.190
--webserver_port=0
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--hive_metastore_uris=thrift://127.0.0.1:43863
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260926 01:56:56.880443 11637 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:56:56.881176 11637 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:56:56.881669 11637 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:56:56.920279 11637 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260926 01:56:56.920591 11637 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260926 01:56:56.920883 11637 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:56:56.921139 11637 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:56:56.921381 11637 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:56:56.921636 11637 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:56:56.962911 11637 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43863
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.190:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.76.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:56:56.964671 11637 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:56:56.966677 11637 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:56:56.980525 11643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:56:58.382120 11642 debug-util.cc:398] Leaking SignalData structure 0x7b08000283c0 after lost signal to thread 11637
W20260926 01:56:58.544338 11637 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.565s user 0.595s sys 0.969s
W20260926 01:56:58.544783 11637 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.565s user 0.595s sys 0.969s
W20260926 01:56:56.980644 11644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:56:58.546089 11645 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1564 milliseconds
W20260926 01:56:58.547194 11646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:56:58.547153 11637 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:56:58.550154 11637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:56:58.552927 11637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:56:58.554301 11637 hybrid_clock.cc:648] HybridClock initialized: now 1790387818554251 us; error 38 us; skew 500 ppm
I20260926 01:56:58.555183 11637 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:56:58.561687 11637 webserver.cc:533] Webserver started at http://127.8.76.190:36691/ using document root <none> and password file <none>
I20260926 01:56:58.562762 11637 fs_manager.cc:362] Metadata directory not provided
I20260926 01:56:58.563001 11637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:56:58.563489 11637 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:56:58.568238 11637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/instance:
uuid: "d9760316a8f74e829b4c7884b875b70e"
format_stamp: "Formatted at 2026-09-26 01:56:58 on dist-test-slave-5pk1"
I20260926 01:56:58.569396 11637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal/instance:
uuid: "d9760316a8f74e829b4c7884b875b70e"
format_stamp: "Formatted at 2026-09-26 01:56:58 on dist-test-slave-5pk1"
I20260926 01:56:58.577262 11637 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.003s
I20260926 01:56:58.583650 11653 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:56:58.584896 11637 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 01:56:58.585264 11637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
uuid: "d9760316a8f74e829b4c7884b875b70e"
format_stamp: "Formatted at 2026-09-26 01:56:58 on dist-test-slave-5pk1"
I20260926 01:56:58.585609 11637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:56:58.650736 11637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:56:58.652549 11637 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:56:58.653066 11637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:56:58.734368 11637 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.190:41989
I20260926 01:56:58.734491 11704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.190:41989 every 8 connection(s)
I20260926 01:56:58.737481 11637 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
I20260926 01:56:58.744549 11705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:56:58.745914 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 11637
I20260926 01:56:58.746531 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal/instance
I20260926 01:56:58.750728 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.189:34661
--webserver_interface=127.8.76.189
--webserver_port=0
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--hive_metastore_uris=thrift://127.0.0.1:43863
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260926 01:56:58.762001 11705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:56:58.797510 11705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } has no permanent_uuid. Determining permanent_uuid...
W20260926 01:56:58.800119 11655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:56:58.805190 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:56:58.854805 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } attempt: 1
W20260926 01:56:58.859797 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:56:58.928239 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } attempt: 2
W20260926 01:56:58.933354 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:56:59.024856 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } attempt: 3
W20260926 01:56:59.029498 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:56:59.107039 11706 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:56:59.107755 11706 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:56:59.108222 11706 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:56:59.144359 11706 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260926 01:56:59.144672 11706 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260926 01:56:59.144944 11706 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:56:59.145197 11706 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:56:59.145452 11706 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:56:59.145692 11706 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:56:59.172910 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } attempt: 4
W20260926 01:56:59.177317 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:56:59.188071 11706 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43863
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.189:34661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.76.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:56:59.189754 11706 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:56:59.191854 11706 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:56:59.204778 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:56:59.476737 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } attempt: 5
W20260926 01:56:59.484220 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:00.031788 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } attempt: 6
W20260926 01:57:00.062495 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:56:59.205559 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:00.573599 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:00.575546 11717 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1365 milliseconds
W20260926 01:57:00.576514 11706 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.371s user 0.526s sys 0.834s
W20260926 01:57:00.576798 11706 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.372s user 0.526s sys 0.834s
I20260926 01:57:00.577039 11706 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:57:00.578226 11706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:57:00.581357 11706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:57:00.582801 11706 hybrid_clock.cc:648] HybridClock initialized: now 1790387820582765 us; error 58 us; skew 500 ppm
I20260926 01:57:00.583860 11706 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:57:00.593052 11706 webserver.cc:533] Webserver started at http://127.8.76.189:35591/ using document root <none> and password file <none>
I20260926 01:57:00.594286 11706 fs_manager.cc:362] Metadata directory not provided
I20260926 01:57:00.594609 11706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:57:00.595175 11706 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:57:00.600582 11706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data/instance:
uuid: "6819a8c0ba75415c9cf089983ff5c191"
format_stamp: "Formatted at 2026-09-26 01:57:00 on dist-test-slave-5pk1"
I20260926 01:57:00.602070 11706 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/wal/instance:
uuid: "6819a8c0ba75415c9cf089983ff5c191"
format_stamp: "Formatted at 2026-09-26 01:57:00 on dist-test-slave-5pk1"
I20260926 01:57:00.612470 11706 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260926 01:57:00.619910 11727 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:00.621477 11706 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260926 01:57:00.621894 11706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/wal
uuid: "6819a8c0ba75415c9cf089983ff5c191"
format_stamp: "Formatted at 2026-09-26 01:57:00 on dist-test-slave-5pk1"
I20260926 01:57:00.622323 11706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:57:00.722046 11706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:57:00.723886 11706 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:57:00.724423 11706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:57:00.808830 11706 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.189:34661
I20260926 01:57:00.808910 11778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.189:34661 every 8 connection(s)
I20260926 01:57:00.811988 11706 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/data/info.pb
I20260926 01:57:00.812768 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 11706
I20260926 01:57:00.813218 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-1/wal/instance
I20260926 01:57:00.817118 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:36403
--webserver_interface=127.8.76.188
--webserver_port=0
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--hive_metastore_uris=thrift://127.0.0.1:43863
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260926 01:57:00.821028 11779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:00.834605 11779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:57:00.859246 11779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:57:00.876605 11779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } has no permanent_uuid. Determining permanent_uuid...
W20260926 01:57:00.879997 11728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:00.885237 11779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:00.934966 11779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 1
W20260926 01:57:00.939927 11779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.008481 11779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 2
W20260926 01:57:01.013458 11779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.105026 11779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 3
W20260926 01:57:01.109856 11779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.123080 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } attempt: 7
I20260926 01:57:01.136148 11705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } has no permanent_uuid. Determining permanent_uuid...
W20260926 01:57:01.143724 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:01.180860 11780 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:57:01.181547 11780 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:57:01.181957 11780 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260926 01:57:01.202203 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 1
W20260926 01:57:01.206945 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:01.219939 11780 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260926 01:57:01.220227 11780 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260926 01:57:01.220512 11780 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:57:01.220765 11780 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:57:01.221020 11780 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:57:01.221277 11780 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:57:01.253412 11779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 4
W20260926 01:57:01.258642 11779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.263705 11780 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43863
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.76.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:57:01.265303 11780 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:57:01.267347 11780 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:57:01.280650 11792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:57:01.288414 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 2
W20260926 01:57:01.295936 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.381500 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 3
W20260926 01:57:01.389055 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.529641 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 4
W20260926 01:57:01.537288 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.558184 11779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 5
W20260926 01:57:01.566959 11779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:01.820930 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 5
W20260926 01:57:01.828703 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:02.114557 11779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 6
W20260926 01:57:02.146103 11779 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:02.381287 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 6
W20260926 01:57:02.412376 11705 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.76.188:36403: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:02.683287 11791 debug-util.cc:398] Leaking SignalData structure 0x7b08000283c0 after lost signal to thread 11780
W20260926 01:57:02.869995 11780 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.589s user 0.643s sys 0.944s
W20260926 01:57:02.870440 11780 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.590s user 0.643s sys 0.944s
W20260926 01:57:01.281394 11793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:02.872603 11795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:02.874934 11794 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1590 milliseconds
I20260926 01:57:02.874974 11780 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:57:02.876312 11780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:57:02.879106 11780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:57:02.880487 11780 hybrid_clock.cc:648] HybridClock initialized: now 1790387822880405 us; error 79 us; skew 500 ppm
I20260926 01:57:02.881368 11780 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:57:02.888405 11780 webserver.cc:533] Webserver started at http://127.8.76.188:46041/ using document root <none> and password file <none>
I20260926 01:57:02.889619 11780 fs_manager.cc:362] Metadata directory not provided
I20260926 01:57:02.889848 11780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:57:02.890339 11780 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:57:02.896134 11780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/instance:
uuid: "596cdfef9f194805948e35ffe299ed65"
format_stamp: "Formatted at 2026-09-26 01:57:02 on dist-test-slave-5pk1"
I20260926 01:57:02.897413 11780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal/instance:
uuid: "596cdfef9f194805948e35ffe299ed65"
format_stamp: "Formatted at 2026-09-26 01:57:02 on dist-test-slave-5pk1"
I20260926 01:57:02.905740 11780 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20260926 01:57:02.912544 11806 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:02.914013 11780 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 01:57:02.914443 11780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
uuid: "596cdfef9f194805948e35ffe299ed65"
format_stamp: "Formatted at 2026-09-26 01:57:02 on dist-test-slave-5pk1"
I20260926 01:57:02.914860 11780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:57:02.963128 11780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:57:02.964926 11780 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:57:02.965402 11780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:57:03.048524 11780 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.188:36403
I20260926 01:57:03.048583 11857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.188:36403 every 8 connection(s)
I20260926 01:57:03.051791 11780 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
I20260926 01:57:03.056901 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 11780
I20260926 01:57:03.057399 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal/instance
I20260926 01:57:03.058413 11858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:03.075461 11858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:57:03.100558 11858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:57:03.111343 11858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } has no permanent_uuid. Determining permanent_uuid...
I20260926 01:57:03.128945 11858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Bootstrap starting.
I20260926 01:57:03.136070 11858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:03.138734 11858 log.cc:826] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Log is configured to *not* fsync() on all Append() calls
I20260926 01:57:03.143954 11858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: No bootstrap required, opened a new log
I20260926 01:57:03.165750 11858 raft_consensus.cc:359] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:03.166535 11858 raft_consensus.cc:385] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:03.166815 11858 raft_consensus.cc:740] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596cdfef9f194805948e35ffe299ed65, State: Initialized, Role: FOLLOWER
I20260926 01:57:03.167683 11858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:03.170568 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.171473 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:03.173492 11858 sys_catalog.cc:565] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:57:03.206686 11779 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 7
W20260926 01:57:03.221289 11812 tablet.cc:2433] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 01:57:03.222393 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.222756 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.237123 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.237675 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.203424 11780 master_runner.cc:187] Error getting master registration for 127.8.76.190:41989: OK, instance_id { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" instance_seqno: 1790387818712367 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:57:03.242888 11780 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:57:03.251955 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.252498 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.264952 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: Bootstrap starting.
W20260926 01:57:03.267019 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.267364 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.274258 11779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:03.277070 11779 log.cc:826] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: Log is configured to *not* fsync() on all Append() calls
W20260926 01:57:03.281919 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.282382 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.287309 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: No bootstrap required, opened a new log
W20260926 01:57:03.297186 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.297674 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.312597 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.313059 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.327884 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.328369 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.334551 11779 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:03.335680 11779 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:03.336074 11779 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6819a8c0ba75415c9cf089983ff5c191, State: Initialized, Role: FOLLOWER
I20260926 01:57:03.337455 11779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
W20260926 01:57:03.343031 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.343436 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.344882 11884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.345884 11884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:03.347178 11779 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 01:57:03.358213 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.358770 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.373715 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.374167 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.390019 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.390473 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.396747 11865 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:03.397380 11865 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:03.404044 11754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "596cdfef9f194805948e35ffe299ed65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
W20260926 01:57:03.405028 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 01:57:03.405138 11754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596cdfef9f194805948e35ffe299ed65 in term 0.
W20260926 01:57:03.405359 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.407302 11808 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191; no voters:
I20260926 01:57:03.409786 11865 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
I20260926 01:57:03.411378 11865 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:57:03.412106 11865 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:57:03.417495 11865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:03.441069 11865 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:03.447906 11705 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } attempt: 7
W20260926 01:57:03.453926 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.455744 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.457242 11754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "596cdfef9f194805948e35ffe299ed65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191"
I20260926 01:57:03.457932 11754 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:03.465744 11754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596cdfef9f194805948e35ffe299ed65 in term 1.
I20260926 01:57:03.467250 11808 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191; no voters:
I20260926 01:57:03.468297 11900 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:57:03.471196 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.471832 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.481468 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.481848 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.486172 11865 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 1 election: Requested vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
W20260926 01:57:03.486907 11733 tablet.cc:2433] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:57:03.493769 11900 raft_consensus.cc:697] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 1 LEADER]: Becoming Leader. State: Replica: 596cdfef9f194805948e35ffe299ed65, State: Running, Role: LEADER
I20260926 01:57:03.495039 11900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
W20260926 01:57:03.505571 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.505946 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.513495 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596cdfef9f194805948e35ffe299ed65. Latest consensus state: current_term: 1 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.514377 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: LEADER
W20260926 01:57:03.518718 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.519603 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.521560 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.521885 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.535168 11879 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.535887 11879 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.536568 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.536885 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.537623 11905 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:57:03.538043 11705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Bootstrap starting.
W20260926 01:57:03.551429 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.551803 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.555302 11705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:03.557267 11905 catalog_manager.cc:1533] Initializing Kudu cluster ID...
W20260926 01:57:03.566290 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.566675 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.569840 11705 log.cc:826] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:57:03.433094 11706 master_runner.cc:187] Error getting master registration for 127.8.76.190:41989: OK, instance_id { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" instance_seqno: 1790387818712367 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 01:57:03.572045 11706 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:57:03.581231 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.581606 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.583062 11705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: No bootstrap required, opened a new log
W20260926 01:57:03.596189 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.596572 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.604032 11754 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Refusing update from remote peer 596cdfef9f194805948e35ffe299ed65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:57:03.605895 11865 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:57:03.627357 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.628525 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.652719 11884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596cdfef9f194805948e35ffe299ed65. Latest consensus state: current_term: 1 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.653554 11884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:03.658864 11908 mvcc.cc:204] Tried to move back new op lower bound from 7333428525463367680 to 7333428525049741312. Current Snapshot: MvccSnapshot[applied={T|T < 7333428525463367680}]
I20260926 01:57:03.670166 11884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.671509 11884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:03.676492 11900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.677223 11900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: LEADER
I20260926 01:57:03.676879 11705 raft_consensus.cc:359] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:03.677748 11705 raft_consensus.cc:385] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:03.678050 11705 raft_consensus.cc:740] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9760316a8f74e829b4c7884b875b70e, State: Initialized, Role: FOLLOWER
I20260926 01:57:03.679078 11705 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:03.683701 11905 catalog_manager.cc:1396] Generated new cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:03.684046 11905 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
W20260926 01:57:03.736402 11899 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 01:57:03.738926 11899 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.739737 11912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.745153 11912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:03.751149 11705 sys_catalog.cc:565] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:57:03.751351 11680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "596cdfef9f194805948e35ffe299ed65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:03.752408 11680 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596cdfef9f194805948e35ffe299ed65 in term 0.
I20260926 01:57:03.755153 11678 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:03.757493 11679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "596cdfef9f194805948e35ffe299ed65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e"
I20260926 01:57:03.779028 11678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Refusing update from remote peer 596cdfef9f194805948e35ffe299ed65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 01:57:03.783506 11899 catalog_manager.cc:1308] Loaded cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:03.787328 11903 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:57:03.814620 11899 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: loading cluster ID for follower catalog manager: success
W20260926 01:57:03.829850 11899 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 01:57:03.833850 11899 catalog_manager.cc:896] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.857613 11899 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 01:57:03.861433 11899 catalog_manager.cc:896] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 01:57:03.887073 11912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596cdfef9f194805948e35ffe299ed65. Latest consensus state: current_term: 1 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.890266 11912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:03.894042 11900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:03.894912 11900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: LEADER
I20260926 01:57:03.905885 11905 catalog_manager.cc:1419] Generated new certificate authority record
I20260926 01:57:03.908442 11905 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:57:03.939311 11899 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: acquiring CA information for follower catalog manager: success
W20260926 01:57:03.945984 11899 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260926 01:57:03.991318 11899 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260926 01:57:04.013568 11905 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Generated new TSK 0
I20260926 01:57:04.015205 11905 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:57:04.020871 11905 catalog_manager.cc:1576] Loading latest processed Hive Metastore notification log event ID...
I20260926 01:57:04.040005 11899 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:57:04.045159 11905 catalog_manager.cc:6068] Last processed Hive Metastore notification event ID: -1
I20260926 01:57:04.089246 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.76.129:0
--local_ip_for_outbound_sockets=127.8.76.129
--webserver_interface=127.8.76.129
--webserver_port=0
--tserver_master_addrs=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260926 01:57:04.521642 11925 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:04.522590 11925 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:04.533597 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:04.535090 11925 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:04.537341 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:04.539608 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:04.540676 11925 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:04.850353 11917 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:04.851413 11917 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: FOLLOWER
W20260926 01:57:04.858975 11914 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.000s sys 0.006s
W20260926 01:57:04.859472 11914 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.958s user 0.000s sys 0.006s
W20260926 01:57:04.863637 11705 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 0.999s user 0.000s sys 0.011s
W20260926 01:57:04.863947 11705 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 0.999s user 0.000s sys 0.011s
I20260926 01:57:04.968304 11917 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:04.968853 11917 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:04.981024 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:04.980095 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:04.978833 11917 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:04.989382 11655 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:04.991948 11917 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:05.015767 11935 catalog_manager.cc:1308] Loaded cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:05.027217 11935 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: loading cluster ID for follower catalog manager: success
I20260926 01:57:05.042802 11935 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: acquiring CA information for follower catalog manager: success
I20260926 01:57:05.065212 11935 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 01:57:05.248558 11924 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:57:05.249434 11924 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:57:05.251232 11924 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:57:05.321208 11924 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:57:05.322759 11924 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.76.129
I20260926 01:57:05.407967 11924 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.76.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.76.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.76.129
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:57:05.409999 11924 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:57:05.412470 11924 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:57:05.454552 11943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:57:05.519254 11948 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:05.520061 11948 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:05.523418 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:05.525875 11948 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:05.528322 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:05.530047 11654 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:05.530833 11948 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:05.615427 11949 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:05.616112 11949 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:05.619436 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:05.620168 11949 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:05.621004 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:05.624817 11730 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:05.625638 11949 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:06.503189 11950 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:06.503819 11950 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:06.507026 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:06.518728 11950 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:06.520314 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:06.522166 11654 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:06.523016 11950 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:07.107635 11952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:07.108457 11952 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:07.111985 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:07.117671 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:07.120719 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:07.134148 11953 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:07.134903 11952 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:06.849789 11941 debug-util.cc:398] Leaking SignalData structure 0x7b0800037180 after lost signal to thread 11924
W20260926 01:57:07.778855 11941 kernel_stack_watchdog.cc:198] Thread 11924 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 388ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260926 01:57:07.779317 11924 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.320s user 0.515s sys 0.818s
W20260926 01:57:07.782878 11924 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.324s user 0.515s sys 0.818s
W20260926 01:57:05.459230 11942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:07.786206 11946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:07.789728 11944 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2326 milliseconds
I20260926 01:57:07.789744 11924 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:57:07.791615 11924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:57:07.804577 11924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:57:07.814668 11924 hybrid_clock.cc:648] HybridClock initialized: now 1790387827814563 us; error 80 us; skew 500 ppm
I20260926 01:57:07.816033 11924 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:57:07.848706 11924 webserver.cc:533] Webserver started at http://127.8.76.129:37853/ using document root <none> and password file <none>
I20260926 01:57:07.850215 11924 fs_manager.cc:362] Metadata directory not provided
I20260926 01:57:07.850544 11924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:57:07.851214 11924 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:57:07.857998 11924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data/instance:
uuid: "5d58342c1ed14ef9a6815b9b5a219b56"
format_stamp: "Formatted at 2026-09-26 01:57:07 on dist-test-slave-5pk1"
I20260926 01:57:07.859647 11924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/instance:
uuid: "5d58342c1ed14ef9a6815b9b5a219b56"
format_stamp: "Formatted at 2026-09-26 01:57:07 on dist-test-slave-5pk1"
I20260926 01:57:07.869839 11924 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260926 01:57:07.877175 11958 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:07.878717 11924 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260926 01:57:07.879290 11924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal
uuid: "5d58342c1ed14ef9a6815b9b5a219b56"
format_stamp: "Formatted at 2026-09-26 01:57:07 on dist-test-slave-5pk1"
I20260926 01:57:07.879925 11924 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:57:07.947496 11924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:57:07.949486 11924 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:57:07.949973 11924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:57:07.953311 11924 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260926 01:57:07.958791 11924 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:57:07.959000 11924 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:07.959286 11924 ts_tablet_manager.cc:616] Registered 0 tablets
I20260926 01:57:07.959435 11924 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:08.158417 11924 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.129:41369
I20260926 01:57:08.158572 12070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.129:41369 every 8 connection(s)
I20260926 01:57:08.161414 11924 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/data/info.pb
I20260926 01:57:08.163427 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 11924
I20260926 01:57:08.164341 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/instance
I20260926 01:57:08.189055 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.76.130:0
--local_ip_for_outbound_sockets=127.8.76.130
--webserver_interface=127.8.76.130
--webserver_port=0
--tserver_master_addrs=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260926 01:57:08.259830 12071 heartbeater.cc:344] Connected to a master server at 127.8.76.188:36403
I20260926 01:57:08.260473 12071 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:08.261852 12072 heartbeater.cc:344] Connected to a master server at 127.8.76.189:34661
I20260926 01:57:08.262231 12072 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:08.262167 12071 heartbeater.cc:507] Master 127.8.76.188:36403 requested a full tablet report, sending...
I20260926 01:57:08.263248 12072 heartbeater.cc:507] Master 127.8.76.189:34661 requested a full tablet report, sending...
I20260926 01:57:08.266732 11823 ts_manager.cc:194] Registered new tserver with Master: 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369)
I20260926 01:57:08.266901 11744 ts_manager.cc:194] Registered new tserver with Master: 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369)
I20260926 01:57:08.268373 12075 heartbeater.cc:344] Connected to a master server at 127.8.76.190:41989
I20260926 01:57:08.268765 12075 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:08.269672 12075 heartbeater.cc:507] Master 127.8.76.190:41989 requested a full tablet report, sending...
I20260926 01:57:08.269892 11823 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.76.129:41757
I20260926 01:57:08.273693 11669 ts_manager.cc:194] Registered new tserver with Master: 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369)
I20260926 01:57:08.467885 12082 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:08.468609 12082 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:08.473198 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:08.473201 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:08.475438 11655 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:08.475665 12082 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:08.476501 12082 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:57:08.603369 12076 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:57:08.603966 12076 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:57:08.604566 12076 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:57:08.642983 12076 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:57:08.643996 12076 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.76.130
I20260926 01:57:08.687904 12076 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.76.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.76.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.76.130
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:57:08.689765 12076 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:57:08.691920 12076 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:57:08.708729 12088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:57:08.774678 12071 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:09.969070 12095 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:09.970201 12095 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:09.977205 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:09.982322 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:09.986881 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:09.993860 12095 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:09.996903 12095 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:10.033914 12097 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:10.035300 12097 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:10.039964 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:10.050827 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:10.056833 11654 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:10.072669 12097 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:10.074201 12097 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:57:08.711644 12089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:10.110512 12087 debug-util.cc:398] Leaking SignalData structure 0x7b0800037180 after lost signal to thread 12076
W20260926 01:57:10.335029 12087 kernel_stack_watchdog.cc:198] Thread 12076 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260926 01:57:10.336066 12076 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.628s user 0.643s sys 0.937s
W20260926 01:57:10.336467 12076 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.628s user 0.643s sys 0.938s
W20260926 01:57:10.336676 12091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:10.342978 12090 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1631 milliseconds
I20260926 01:57:10.343065 12076 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:57:10.344432 12076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:57:10.346748 12076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:57:10.348153 12076 hybrid_clock.cc:648] HybridClock initialized: now 1790387830348125 us; error 52 us; skew 500 ppm
I20260926 01:57:10.348984 12076 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:57:10.355899 12076 webserver.cc:533] Webserver started at http://127.8.76.130:35391/ using document root <none> and password file <none>
I20260926 01:57:10.356998 12076 fs_manager.cc:362] Metadata directory not provided
I20260926 01:57:10.357213 12076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:57:10.357647 12076 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:57:10.362323 12076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data/instance:
uuid: "1b877862e4a543229352921130f57068"
format_stamp: "Formatted at 2026-09-26 01:57:10 on dist-test-slave-5pk1"
I20260926 01:57:10.363749 12076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/instance:
uuid: "1b877862e4a543229352921130f57068"
format_stamp: "Formatted at 2026-09-26 01:57:10 on dist-test-slave-5pk1"
I20260926 01:57:10.372671 12076 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260926 01:57:10.380429 12103 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:10.381718 12076 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260926 01:57:10.382082 12076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal
uuid: "1b877862e4a543229352921130f57068"
format_stamp: "Formatted at 2026-09-26 01:57:10 on dist-test-slave-5pk1"
I20260926 01:57:10.382445 12076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:57:10.439211 12076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:57:10.441138 12076 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:57:10.441574 12076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:57:10.444577 12076 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260926 01:57:10.449275 12076 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:57:10.449483 12076 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:10.449704 12076 ts_tablet_manager.cc:616] Registered 0 tablets
I20260926 01:57:10.449836 12076 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:10.611214 12076 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.130:34281
I20260926 01:57:10.611316 12215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.130:34281 every 8 connection(s)
I20260926 01:57:10.614255 12076 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/data/info.pb
I20260926 01:57:10.621150 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 12076
I20260926 01:57:10.621762 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/instance
I20260926 01:57:10.653988 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.76.131:0
--local_ip_for_outbound_sockets=127.8.76.131
--webserver_interface=127.8.76.131
--webserver_port=0
--tserver_master_addrs=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260926 01:57:10.674947 12217 heartbeater.cc:344] Connected to a master server at 127.8.76.189:34661
I20260926 01:57:10.675585 12217 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:10.676416 12216 heartbeater.cc:344] Connected to a master server at 127.8.76.188:36403
I20260926 01:57:10.676829 12216 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:10.677183 12217 heartbeater.cc:507] Master 127.8.76.189:34661 requested a full tablet report, sending...
I20260926 01:57:10.677882 12216 heartbeater.cc:507] Master 127.8.76.188:36403 requested a full tablet report, sending...
I20260926 01:57:10.680969 11744 ts_manager.cc:194] Registered new tserver with Master: 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:10.682582 12218 heartbeater.cc:344] Connected to a master server at 127.8.76.190:41989
I20260926 01:57:10.682492 11823 ts_manager.cc:194] Registered new tserver with Master: 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:10.682895 12218 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:10.683805 12218 heartbeater.cc:507] Master 127.8.76.190:41989 requested a full tablet report, sending...
I20260926 01:57:10.684474 11823 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.76.130:50843
I20260926 01:57:10.686367 11669 ts_manager.cc:194] Registered new tserver with Master: 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:10.908929 12227 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:10.909418 12227 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:10.911433 12227 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:10.912454 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:10.912756 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:10.914207 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:10.915134 12227 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:57:11.029472 12226 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:57:11.030092 12226 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:57:11.030730 12226 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:57:11.081082 12226 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:57:11.082127 12226 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.76.131
I20260926 01:57:11.100322 12232 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:11.100903 12232 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:11.102727 12232 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:11.103713 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:11.103956 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:11.105927 11654 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:11.106954 12232 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:11.127342 12226 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.76.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.76.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.76.131
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:57:11.128902 12226 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:57:11.130963 12226 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:57:11.145807 12234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:57:11.189215 12216 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:11.472486 12241 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:11.472942 12241 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:11.475822 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:11.476300 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:11.478199 12241 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:11.478199 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:11.479343 12241 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:11.691147 12242 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:11.691790 12242 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:11.694344 12242 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:11.696347 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:11.698767 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:11.702847 11655 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:11.704071 12242 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:12.022904 12243 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:12.023670 12243 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:12.026572 12243 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:12.027798 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:12.031391 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:12.032975 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:12.034373 12243 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:57:11.146878 12235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:12.431310 12236 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260926 01:57:12.437614 12237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:12.438203 12226 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.292s user 0.003s sys 0.003s
W20260926 01:57:12.438536 12226 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.293s user 0.005s sys 0.003s
I20260926 01:57:12.438768 12226 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:57:12.439960 12226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:57:12.442971 12226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:57:12.444373 12226 hybrid_clock.cc:648] HybridClock initialized: now 1790387832444334 us; error 57 us; skew 500 ppm
I20260926 01:57:12.445338 12226 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:57:12.454746 12226 webserver.cc:533] Webserver started at http://127.8.76.131:39411/ using document root <none> and password file <none>
I20260926 01:57:12.455929 12226 fs_manager.cc:362] Metadata directory not provided
I20260926 01:57:12.456203 12226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:57:12.456712 12226 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 01:57:12.461956 12226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data/instance:
uuid: "495ab022dafc493395ae0e21d9a77ee3"
format_stamp: "Formatted at 2026-09-26 01:57:12 on dist-test-slave-5pk1"
I20260926 01:57:12.463444 12226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/instance:
uuid: "495ab022dafc493395ae0e21d9a77ee3"
format_stamp: "Formatted at 2026-09-26 01:57:12 on dist-test-slave-5pk1"
I20260926 01:57:12.473264 12226 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260926 01:57:12.480435 12248 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:12.482090 12226 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 01:57:12.482492 12226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal
uuid: "495ab022dafc493395ae0e21d9a77ee3"
format_stamp: "Formatted at 2026-09-26 01:57:12 on dist-test-slave-5pk1"
I20260926 01:57:12.482968 12226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:57:12.551524 12226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:57:12.553418 12226 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:57:12.553900 12226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:57:12.556833 12226 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260926 01:57:12.561782 12226 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 01:57:12.562018 12226 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:12.562268 12226 ts_tablet_manager.cc:616] Registered 0 tablets
I20260926 01:57:12.562422 12226 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:12.659571 12317 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:12.660166 12317 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:12.662771 12317 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:12.664350 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:12.664342 11833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
I20260926 01:57:12.666435 11654 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:12.668207 12317 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 01:57:12.725757 12226 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.131:46707
I20260926 01:57:12.725899 12361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.131:46707 every 8 connection(s)
I20260926 01:57:12.728739 12226 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/data/info.pb
I20260926 01:57:12.734861 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 12226
I20260926 01:57:12.735519 8498 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/instance
I20260926 01:57:12.779198 12362 heartbeater.cc:344] Connected to a master server at 127.8.76.188:36403
I20260926 01:57:12.779830 12362 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:12.781389 12362 heartbeater.cc:507] Master 127.8.76.188:36403 requested a full tablet report, sending...
I20260926 01:57:12.784685 12363 heartbeater.cc:344] Connected to a master server at 127.8.76.189:34661
I20260926 01:57:12.785080 12363 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:12.785413 11823 ts_manager.cc:194] Registered new tserver with Master: 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:12.786062 12363 heartbeater.cc:507] Master 127.8.76.189:34661 requested a full tablet report, sending...
I20260926 01:57:12.787029 12364 heartbeater.cc:344] Connected to a master server at 127.8.76.190:41989
I20260926 01:57:12.787389 12364 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:12.787482 11823 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.76.131:56531
I20260926 01:57:12.788296 12364 heartbeater.cc:507] Master 127.8.76.190:41989 requested a full tablet report, sending...
I20260926 01:57:12.789599 11743 ts_manager.cc:194] Registered new tserver with Master: 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:12.792585 11670 ts_manager.cc:194] Registered new tserver with Master: 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:12.796592 8498 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260926 01:57:12.859822 11823 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55644:
name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:12.872328 11822 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55644:
name: "default.Table_7440abc68fd740d49170b27f2cb1c731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:57:12.884891 11823 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_47c5a1843d9a4eb2ab8eba451134b168 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:57:12.900750 11820 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55644:
name: "default.Table_5bdf5515f5af48e1bd0f78249a93cedc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:12.887661 11821 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55644:
name: "default.Table_9ecb0b51e0e947d4bc9d0fb60139994f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:57:12.895857 11822 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7440abc68fd740d49170b27f2cb1c731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:57:12.941922 8498 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu with pid 11780
W20260926 01:57:12.986121 12372 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.188:36403)
I20260926 01:57:13.065649 12397 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:13.066476 12397 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:13.075196 12397 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:13.077147 11678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
W20260926 01:57:13.108285 11728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260926 01:57:13.113528 11728 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:13.113935 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:13.114634 12397 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260926 01:57:13.132885 11701 debug-util.cc:398] Leaking SignalData structure 0x7b08000a2c20 after lost signal to thread 11638
W20260926 01:57:13.232394 12216 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:13.274981 12401 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:13.282385 12401 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:13.292429 12362 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:13.294812 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191" is_pre_election: true
I20260926 01:57:13.295475 11753 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9760316a8f74e829b4c7884b875b70e in term 1.
I20260926 01:57:13.296832 11655 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191, d9760316a8f74e829b4c7884b875b70e; no voters:
W20260926 01:57:13.312793 12362 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:13.314502 12403 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 01:57:13.314950 12403 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:13.316454 12401 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:13.323649 11654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260926 01:57:13.326695 12403 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 1 FOLLOWER]: Advancing to term 2
I20260926 01:57:13.336591 12403 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
W20260926 01:57:13.350289 11654 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:13.352365 12403 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 election: Requested vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:13.352325 11753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6819a8c0ba75415c9cf089983ff5c191"
I20260926 01:57:13.352947 11753 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 1 FOLLOWER]: Advancing to term 2
W20260926 01:57:13.372089 11654 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:13.372610 11753 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9760316a8f74e829b4c7884b875b70e in term 2.
I20260926 01:57:13.374541 11655 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191, d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65
I20260926 01:57:13.377904 12403 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 FOLLOWER]: Leader election won for term 2
I20260926 01:57:13.378443 12403 raft_consensus.cc:697] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 LEADER]: Becoming Leader. State: Replica: d9760316a8f74e829b4c7884b875b70e, State: Running, Role: LEADER
I20260926 01:57:13.379824 12403 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
W20260926 01:57:13.387728 12071 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:13.397528 12401 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9760316a8f74e829b4c7884b875b70e. Latest consensus state: current_term: 2 leader_uuid: "d9760316a8f74e829b4c7884b875b70e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:13.398378 12401 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: LEADER
I20260926 01:57:13.402762 12408 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:57:13.412097 12408 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:57:13.416121 12408 catalog_manager.cc:1308] Loaded cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:13.416400 12408 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:57:13.424741 12408 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:57:13.429302 12408 catalog_manager.cc:6230] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Loaded TSK: 0
I20260926 01:57:13.430884 12408 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:57:13.435326 12408 catalog_manager.cc:1576] Loading latest processed Hive Metastore notification log event ID...
I20260926 01:57:13.439744 12408 catalog_manager.cc:6068] Last processed Hive Metastore notification event ID: -1
I20260926 01:57:13.467970 11669 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_9ecb0b51e0e947d4bc9d0fb60139994f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:13.468783 11668 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_7440abc68fd740d49170b27f2cb1c731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:13.469664 11667 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:13.460070 11670 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_5bdf5515f5af48e1bd0f78249a93cedc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 01:57:13.526177 11669 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9ecb0b51e0e947d4bc9d0fb60139994f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:57:13.542177 11668 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7440abc68fd740d49170b27f2cb1c731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:57:13.546746 11667 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_47c5a1843d9a4eb2ab8eba451134b168 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 01:57:13.550416 11670 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5bdf5515f5af48e1bd0f78249a93cedc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 01:57:13.865820 11669 catalog_manager.cc:2213] created HMS catalog entry for table [id=c3c32452d955463485f7f60723c019ec]
I20260926 01:57:13.892028 11753 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Refusing update from remote peer d9760316a8f74e829b4c7884b875b70e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260926 01:57:13.894937 12401 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260926 01:57:13.931167 11654 consensus_peers.cc:597] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e -> Peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Couldn't send request to peer 596cdfef9f194805948e35ffe299ed65. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:57:13.933209 12414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9760316a8f74e829b4c7884b875b70e. Latest consensus state: current_term: 2 leader_uuid: "d9760316a8f74e829b4c7884b875b70e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:13.934072 12414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:13.938858 12411 mvcc.cc:204] Tried to move back new op lower bound from 7333428567573123072 to 7333428565535363072. Current Snapshot: MvccSnapshot[applied={T|T < 7333428567573123072}]
I20260926 01:57:13.939287 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:36403
--webserver_interface=127.8.76.188
--webserver_port=46041
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--hive_metastore_uris=thrift://127.0.0.1:43863
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260926 01:57:13.951862 12414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d9760316a8f74e829b4c7884b875b70e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:13.953868 12414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:13.959607 12403 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d9760316a8f74e829b4c7884b875b70e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:13.960403 12403 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: LEADER
I20260926 01:57:13.974754 11670 catalog_manager.cc:2213] created HMS catalog entry for table [id=85a53b32a7774185b1152340e907f122]
I20260926 01:57:13.985378 11669 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43138:
table { table_name: "default.Table_9ecb0b51e0e947d4bc9d0fb60139994f" } new_table_name: "default.Table_75ff2d45355c4608803c3287943fdb45" modify_external_catalogs: true
I20260926 01:57:14.030663 11670 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43138:
table { table_name: "default.Table_5bdf5515f5af48e1bd0f78249a93cedc" } new_table_name: "default.Table_9696800ab3c24a7a911de7daea878289" modify_external_catalogs: true
I20260926 01:57:14.115556 12005 tablet_service.cc:1511] Processing CreateTablet for tablet c34c5cb77d164ddeb904c3b54aba569e (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.117719 12005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34c5cb77d164ddeb904c3b54aba569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.118592 12295 tablet_service.cc:1511] Processing CreateTablet for tablet c34c5cb77d164ddeb904c3b54aba569e (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.120855 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34c5cb77d164ddeb904c3b54aba569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.114447 12006 tablet_service.cc:1511] Processing CreateTablet for tablet 68a9475fbf434bd688e2860929764a6f (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.122946 12296 tablet_service.cc:1511] Processing CreateTablet for tablet 68a9475fbf434bd688e2860929764a6f (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.123672 12006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a9475fbf434bd688e2860929764a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.124142 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a9475fbf434bd688e2860929764a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.143672 11667 catalog_manager.cc:2213] created HMS catalog entry for table [id=8077aa6cba9a4d108d72edd3d90d4236]
I20260926 01:57:14.142968 12151 tablet_service.cc:1511] Processing CreateTablet for tablet 68a9475fbf434bd688e2860929764a6f (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.145239 12151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a9475fbf434bd688e2860929764a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.146349 12150 tablet_service.cc:1511] Processing CreateTablet for tablet c34c5cb77d164ddeb904c3b54aba569e (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.176206 11667 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43138:
table { table_name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168" } modify_external_catalogs: true
I20260926 01:57:14.177031 11667 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43138:
table { table_name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168" } modify_external_catalogs: true
I20260926 01:57:14.183749 11666 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_09a88a4aa04c44fd9ff3aac5e421f67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:14.187647 12431 tablet_bootstrap.cc:492] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:14.197810 12431 tablet_bootstrap.cc:654] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.200656 12431 log.cc:826] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Log is configured to *not* fsync() on all Append() calls
I20260926 01:57:14.204941 12296 tablet_service.cc:1511] Processing CreateTablet for tablet 10a5289af7464d959096cdaaca6d12e0 (DEFAULT_TABLE table=default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.205937 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34c5cb77d164ddeb904c3b54aba569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.205732 12149 tablet_service.cc:1511] Processing CreateTablet for tablet 10a5289af7464d959096cdaaca6d12e0 (DEFAULT_TABLE table=default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.206209 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a5289af7464d959096cdaaca6d12e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.207046 12149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a5289af7464d959096cdaaca6d12e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.211671 12004 tablet_service.cc:1511] Processing CreateTablet for tablet 10a5289af7464d959096cdaaca6d12e0 (DEFAULT_TABLE table=default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.218326 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a5289af7464d959096cdaaca6d12e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.221388 11665 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_5c81b19f115f450fb219ec996c8c470b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:14.231742 12218 heartbeater.cc:499] Master 127.8.76.190:41989 was elected leader, sending a full tablet report...
I20260926 01:57:14.242144 12431 tablet_bootstrap.cc:492] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:14.242884 12431 ts_tablet_manager.cc:1403] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.056s user 0.015s sys 0.008s
I20260926 01:57:14.252583 11664 master_service.cc:960] ReplaceTablet: received request to replace tablet 68a9475fbf434bd688e2860929764a6f from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:14.254664 11663 master_service.cc:960] ReplaceTablet: received request to replace tablet c34c5cb77d164ddeb904c3b54aba569e from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:14.264827 12433 tablet_bootstrap.cc:492] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:14.277310 12433 tablet_bootstrap.cc:654] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.287127 11663 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Sending DeleteTablet for 3 replicas of tablet c34c5cb77d164ddeb904c3b54aba569e
I20260926 01:57:14.289623 11663 catalog_manager.cc:6934] ReplaceTablet: tablet c34c5cb77d164ddeb904c3b54aba569e deleted and replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:14.290900 12148 tablet_service.cc:1558] Processing DeleteTablet for tablet c34c5cb77d164ddeb904c3b54aba569e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.290792 12296 tablet_service.cc:1558] Processing DeleteTablet for tablet c34c5cb77d164ddeb904c3b54aba569e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:14.306783 12364 heartbeater.cc:499] Master 127.8.76.190:41989 was elected leader, sending a full tablet report...
I20260926 01:57:14.307679 11664 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Sending DeleteTablet for 3 replicas of tablet 68a9475fbf434bd688e2860929764a6f
I20260926 01:57:14.322985 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 68a9475fbf434bd688e2860929764a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 1d661349b7d54176ad406fd43872b8dd) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:14.323130 11664 catalog_manager.cc:6934] ReplaceTablet: tablet 68a9475fbf434bd688e2860929764a6f deleted and replaced by tablet 1d661349b7d54176ad406fd43872b8dd
I20260926 01:57:14.325628 12147 tablet_service.cc:1558] Processing DeleteTablet for tablet 68a9475fbf434bd688e2860929764a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 1d661349b7d54176ad406fd43872b8dd) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.315407 12004 tablet_service.cc:1558] Processing DeleteTablet for tablet c34c5cb77d164ddeb904c3b54aba569e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:14.323879 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet 68a9475fbf434bd688e2860929764a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 1d661349b7d54176ad406fd43872b8dd) from {username='slave'} at 127.0.0.1:49028
W20260926 01:57:14.338253 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet 68a9475fbf434bd688e2860929764a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68a9475fbf434bd688e2860929764a6f already in progress: creating tablet
I20260926 01:57:14.338800 12433 log.cc:826] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Log is configured to *not* fsync() on all Append() calls
W20260926 01:57:14.342237 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet c34c5cb77d164ddeb904c3b54aba569e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c34c5cb77d164ddeb904c3b54aba569e already in progress: creating tablet
I20260926 01:57:14.358747 12437 tablet_bootstrap.cc:492] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:14.363868 12433 tablet_bootstrap.cc:492] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:14.364609 12433 ts_tablet_manager.cc:1403] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.100s user 0.013s sys 0.005s
W20260926 01:57:14.365000 11654 catalog_manager.cc:5043] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): delete failed for tablet c34c5cb77d164ddeb904c3b54aba569e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c34c5cb77d164ddeb904c3b54aba569e already in progress: creating tablet
I20260926 01:57:14.369128 12004 tablet_service.cc:1511] Processing CreateTablet for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.371803 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ab9e06a641740c7bbb1afbe0f4943a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:57:14.375487 11654 catalog_manager.cc:5043] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): delete failed for tablet c34c5cb77d164ddeb904c3b54aba569e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c34c5cb77d164ddeb904c3b54aba569e already in progress: creating tablet
I20260926 01:57:14.377229 11662 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_df54eb8e69094653a7e2b65f39cea88c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:14.379447 12437 tablet_bootstrap.cc:654] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.380349 12075 heartbeater.cc:499] Master 127.8.76.190:41989 was elected leader, sending a full tablet report...
W20260926 01:57:14.378667 11654 catalog_manager.cc:5043] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): delete failed for tablet 68a9475fbf434bd688e2860929764a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68a9475fbf434bd688e2860929764a6f already in progress: creating tablet
W20260926 01:57:14.390980 11654 catalog_manager.cc:5043] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): delete failed for tablet 68a9475fbf434bd688e2860929764a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68a9475fbf434bd688e2860929764a6f already in progress: creating tablet
I20260926 01:57:14.396262 12296 tablet_service.cc:1511] Processing CreateTablet for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.397558 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ab9e06a641740c7bbb1afbe0f4943a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.401386 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet c34c5cb77d164ddeb904c3b54aba569e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:14.403934 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet 68a9475fbf434bd688e2860929764a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 1d661349b7d54176ad406fd43872b8dd) from {username='slave'} at 127.0.0.1:49028
W20260926 01:57:14.404028 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet c34c5cb77d164ddeb904c3b54aba569e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c34c5cb77d164ddeb904c3b54aba569e already in progress: creating tablet
I20260926 01:57:14.394304 12151 tablet_service.cc:1511] Processing CreateTablet for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.405915 12151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ab9e06a641740c7bbb1afbe0f4943a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:57:14.405905 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet 68a9475fbf434bd688e2860929764a6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68a9475fbf434bd688e2860929764a6f already in progress: creating tablet
I20260926 01:57:14.417444 12437 log.cc:826] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Log is configured to *not* fsync() on all Append() calls
I20260926 01:57:14.421248 12295 tablet_service.cc:1511] Processing CreateTablet for tablet 1d661349b7d54176ad406fd43872b8dd (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.422750 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d661349b7d54176ad406fd43872b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.419044 12150 tablet_service.cc:1511] Processing CreateTablet for tablet 1d661349b7d54176ad406fd43872b8dd (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.425339 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d661349b7d54176ad406fd43872b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.447432 12004 tablet_service.cc:1511] Processing CreateTablet for tablet 1d661349b7d54176ad406fd43872b8dd (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.449302 12431 raft_consensus.cc:359] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } }
I20260926 01:57:14.453727 12431 raft_consensus.cc:385] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.454210 12431 raft_consensus.cc:740] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.455545 12431 consensus_queue.cc:260] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } }
I20260926 01:57:14.456476 12437 tablet_bootstrap.cc:492] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:14.457167 12437 ts_tablet_manager.cc:1403] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.099s user 0.014s sys 0.004s
I20260926 01:57:14.471444 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d661349b7d54176ad406fd43872b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.473393 12431 ts_tablet_manager.cc:1434] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.230s user 0.036s sys 0.008s
I20260926 01:57:14.485342 12431 tablet_bootstrap.cc:492] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:14.495332 12296 tablet_service.cc:1558] Processing DeleteTablet for tablet 68a9475fbf434bd688e2860929764a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 1d661349b7d54176ad406fd43872b8dd) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:14.496361 12436 tablet_replica.cc:333] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
I20260926 01:57:14.497239 12436 raft_consensus.cc:2243] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.497852 12436 raft_consensus.cc:2272] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:14.504951 12431 tablet_bootstrap.cc:654] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.520823 12436 ts_tablet_manager.cc:1916] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.525432 12437 raft_consensus.cc:359] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } }
I20260926 01:57:14.534665 12437 raft_consensus.cc:385] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.535190 12437 raft_consensus.cc:740] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.536659 12437 consensus_queue.cc:260] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } }
I20260926 01:57:14.541981 11661 master_service.cc:960] ReplaceTablet: received request to replace tablet 10a5289af7464d959096cdaaca6d12e0 from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:14.543087 11663 master_service.cc:960] ReplaceTablet: received request to replace tablet c34c5cb77d164ddeb904c3b54aba569e from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:14.554847 12437 ts_tablet_manager.cc:1434] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.092s user 0.036s sys 0.000s
I20260926 01:57:14.556298 12437 tablet_bootstrap.cc:492] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:14.563280 12437 tablet_bootstrap.cc:654] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.572088 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 68a9475fbf434bd688e2860929764a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 1d661349b7d54176ad406fd43872b8dd) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.573232 12435 tablet_replica.cc:333] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:14.574167 12435 raft_consensus.cc:2243] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.574864 12435 raft_consensus.cc:2272] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:14.578145 12435 ts_tablet_manager.cc:1916] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.588881 11661 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Sending DeleteTablet for 3 replicas of tablet 10a5289af7464d959096cdaaca6d12e0
I20260926 01:57:14.590785 11661 catalog_manager.cc:6934] ReplaceTablet: tablet 10a5289af7464d959096cdaaca6d12e0 deleted and replaced by tablet b554bcce374d4e5fa02957cd8aaebde8
I20260926 01:57:14.591463 12004 tablet_service.cc:1558] Processing DeleteTablet for tablet 10a5289af7464d959096cdaaca6d12e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b554bcce374d4e5fa02957cd8aaebde8) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:14.593340 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 10a5289af7464d959096cdaaca6d12e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b554bcce374d4e5fa02957cd8aaebde8) from {username='slave'} at 127.0.0.1:59244
W20260926 01:57:14.593587 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet 10a5289af7464d959096cdaaca6d12e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10a5289af7464d959096cdaaca6d12e0 already in progress: creating tablet
I20260926 01:57:14.603237 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 10a5289af7464d959096cdaaca6d12e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b554bcce374d4e5fa02957cd8aaebde8) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.617029 12437 tablet_bootstrap.cc:492] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:14.617756 12437 ts_tablet_manager.cc:1403] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.007s
I20260926 01:57:14.620898 12437 raft_consensus.cc:359] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.632488 12437 raft_consensus.cc:385] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.632944 12437 raft_consensus.cc:740] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.637037 12433 raft_consensus.cc:359] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.639492 12437 consensus_queue.cc:260] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.639876 12004 tablet_service.cc:1511] Processing CreateTablet for tablet b554bcce374d4e5fa02957cd8aaebde8 (DEFAULT_TABLE table=default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.640784 12150 tablet_service.cc:1511] Processing CreateTablet for tablet b554bcce374d4e5fa02957cd8aaebde8 (DEFAULT_TABLE table=default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.642896 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b554bcce374d4e5fa02957cd8aaebde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.642892 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b554bcce374d4e5fa02957cd8aaebde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.649585 12295 tablet_service.cc:1511] Processing CreateTablet for tablet b554bcce374d4e5fa02957cd8aaebde8 (DEFAULT_TABLE table=default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.650790 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b554bcce374d4e5fa02957cd8aaebde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.655017 12437 ts_tablet_manager.cc:1434] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20260926 01:57:14.650962 12431 tablet_bootstrap.cc:492] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:14.664449 12433 raft_consensus.cc:385] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.664922 12433 raft_consensus.cc:740] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.667093 12151 tablet_service.cc:1558] Processing DeleteTablet for tablet c34c5cb77d164ddeb904c3b54aba569e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.667306 12437 tablet_bootstrap.cc:492] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:14.669523 12435 ts_tablet_manager.cc:1929] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:14.669705 12433 consensus_queue.cc:260] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.675139 12435 log.cc:1199] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/68a9475fbf434bd688e2860929764a6f
I20260926 01:57:14.678249 12431 ts_tablet_manager.cc:1403] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.193s user 0.013s sys 0.000s
I20260926 01:57:14.683353 12435 ts_tablet_manager.cc:1950] T 68a9475fbf434bd688e2860929764a6f P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:14.694825 12431 raft_consensus.cc:359] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.696525 11654 catalog_manager.cc:5063] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): tablet 68a9475fbf434bd688e2860929764a6f (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]) successfully deleted
I20260926 01:57:14.698388 12431 raft_consensus.cc:385] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.698740 12431 raft_consensus.cc:740] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.697968 12436 ts_tablet_manager.cc:1929] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:14.699679 12436 log.cc:1199] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/68a9475fbf434bd688e2860929764a6f
I20260926 01:57:14.686915 12437 tablet_bootstrap.cc:654] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.702069 12436 ts_tablet_manager.cc:1950] T 68a9475fbf434bd688e2860929764a6f P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
W20260926 01:57:14.703682 11654 catalog_manager.cc:5043] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): delete failed for tablet 10a5289af7464d959096cdaaca6d12e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10a5289af7464d959096cdaaca6d12e0 already in progress: creating tablet
I20260926 01:57:14.700250 12431 consensus_queue.cc:260] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.708854 12431 ts_tablet_manager.cc:1434] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260926 01:57:14.710109 12435 tablet_replica.cc:333] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:14.711128 12435 raft_consensus.cc:2243] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.712652 12435 raft_consensus.cc:2272] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:14.716962 12296 tablet_service.cc:1558] Processing DeleteTablet for tablet c34c5cb77d164ddeb904c3b54aba569e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:14.718149 12435 ts_tablet_manager.cc:1916] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.720702 11654 catalog_manager.cc:5063] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): tablet 68a9475fbf434bd688e2860929764a6f (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]) successfully deleted
I20260926 01:57:14.721186 12436 tablet_replica.cc:333] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
W20260926 01:57:14.725558 11654 catalog_manager.cc:5043] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): delete failed for tablet 10a5289af7464d959096cdaaca6d12e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10a5289af7464d959096cdaaca6d12e0 already in progress: creating tablet
I20260926 01:57:14.733250 12433 ts_tablet_manager.cc:1434] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.346s user 0.032s sys 0.004s
I20260926 01:57:14.723734 12431 tablet_bootstrap.cc:492] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:14.736354 12437 tablet_bootstrap.cc:492] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:14.736929 12437 ts_tablet_manager.cc:1403] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.007s
I20260926 01:57:14.739996 12437 raft_consensus.cc:359] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.741252 12437 raft_consensus.cc:385] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.741591 12437 raft_consensus.cc:740] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.742600 12437 consensus_queue.cc:260] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.721886 12436 raft_consensus.cc:2243] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.747103 12436 raft_consensus.cc:2272] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260926 01:57:14.752478 12443 raft_consensus.cc:670] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20260926 01:57:14.754848 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 10a5289af7464d959096cdaaca6d12e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b554bcce374d4e5fa02957cd8aaebde8) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.760669 11664 master_service.cc:960] ReplaceTablet: received request to replace tablet 5ab9e06a641740c7bbb1afbe0f4943a0 from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:14.762552 12436 ts_tablet_manager.cc:1916] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.764506 12433 tablet_bootstrap.cc:492] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:14.763789 12437 ts_tablet_manager.cc:1434] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260926 01:57:14.768851 12435 ts_tablet_manager.cc:1929] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:14.769418 12435 log.cc:1199] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/c34c5cb77d164ddeb904c3b54aba569e
I20260926 01:57:14.770198 12435 ts_tablet_manager.cc:1950] T c34c5cb77d164ddeb904c3b54aba569e P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:14.771965 12437 tablet_bootstrap.cc:492] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:14.773000 12435 tablet_replica.cc:333] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:14.773263 11654 catalog_manager.cc:5063] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): tablet c34c5cb77d164ddeb904c3b54aba569e (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:14.773677 12435 raft_consensus.cc:2243] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.774195 12435 raft_consensus.cc:2272] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:14.787729 12433 tablet_bootstrap.cc:654] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.788542 12435 ts_tablet_manager.cc:1916] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.793090 11664 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Sending DeleteTablet for 3 replicas of tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:14.796491 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ac6c587687c4d868586da53310dfa5b) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.798692 11664 catalog_manager.cc:6934] ReplaceTablet: tablet 5ab9e06a641740c7bbb1afbe0f4943a0 deleted and replaced by tablet 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:14.801692 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet c34c5cb77d164ddeb904c3b54aba569e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab9e06a641740c7bbb1afbe0f4943a0) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:14.802857 12438 tablet_replica.cc:333] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: stopping tablet replica
I20260926 01:57:14.803784 12438 raft_consensus.cc:2243] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.804462 12438 raft_consensus.cc:2272] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:14.806241 11661 master_service.cc:960] ReplaceTablet: received request to replace tablet 68a9475fbf434bd688e2860929764a6f from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:14.808786 12435 ts_tablet_manager.cc:1929] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:14.809021 12438 ts_tablet_manager.cc:1916] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.809243 12435 log.cc:1199] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/10a5289af7464d959096cdaaca6d12e0
I20260926 01:57:14.809940 12435 ts_tablet_manager.cc:1950] T 10a5289af7464d959096cdaaca6d12e0 P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:14.812871 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ac6c587687c4d868586da53310dfa5b) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:14.812868 11654 catalog_manager.cc:5063] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): tablet 10a5289af7464d959096cdaaca6d12e0 (table default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]) successfully deleted
I20260926 01:57:14.813678 12004 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ac6c587687c4d868586da53310dfa5b) from {username='slave'} at 127.0.0.1:49028
W20260926 01:57:14.813699 11654 catalog_manager.cc:5043] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): delete failed for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ab9e06a641740c7bbb1afbe0f4943a0 already in progress: creating tablet
I20260926 01:57:14.814739 12437 tablet_bootstrap.cc:654] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.829456 12436 ts_tablet_manager.cc:1929] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:14.831694 12436 log.cc:1199] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/c34c5cb77d164ddeb904c3b54aba569e
I20260926 01:57:14.834070 12437 tablet_bootstrap.cc:492] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:14.834761 12437 ts_tablet_manager.cc:1403] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.007s
I20260926 01:57:14.837743 12437 raft_consensus.cc:359] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.830032 12431 tablet_bootstrap.cc:654] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.839362 12436 ts_tablet_manager.cc:1950] T c34c5cb77d164ddeb904c3b54aba569e P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
I20260926 01:57:14.842726 11654 catalog_manager.cc:5063] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): tablet c34c5cb77d164ddeb904c3b54aba569e (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:14.843578 12437 raft_consensus.cc:385] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.844022 12437 raft_consensus.cc:740] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.845038 12437 consensus_queue.cc:260] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:14.847522 11654 catalog_manager.cc:5043] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): delete failed for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ab9e06a641740c7bbb1afbe0f4943a0 already in progress: creating tablet
I20260926 01:57:14.855351 12433 tablet_bootstrap.cc:492] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:14.855890 12433 ts_tablet_manager.cc:1403] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.005s
I20260926 01:57:14.857121 12437 ts_tablet_manager.cc:1434] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260926 01:57:14.860003 12437 tablet_bootstrap.cc:492] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:14.866717 12437 tablet_bootstrap.cc:654] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.868418 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ac6c587687c4d868586da53310dfa5b) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:14.869270 12435 tablet_replica.cc:333] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:14.870003 12435 raft_consensus.cc:2243] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.870586 12435 raft_consensus.cc:2272] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:14.873627 12435 ts_tablet_manager.cc:1916] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.863399 12433 raft_consensus.cc:359] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } }
I20260926 01:57:14.874400 12433 raft_consensus.cc:385] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.874788 12433 raft_consensus.cc:740] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.875568 12433 consensus_queue.cc:260] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } }
I20260926 01:57:14.900332 12004 tablet_service.cc:1558] Processing DeleteTablet for tablet 68a9475fbf434bd688e2860929764a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 1d661349b7d54176ad406fd43872b8dd) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:14.901741 12005 tablet_service.cc:1511] Processing CreateTablet for tablet 6ac6c587687c4d868586da53310dfa5b (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.901506 12431 tablet_bootstrap.cc:492] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:14.903551 12431 ts_tablet_manager.cc:1403] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.180s user 0.013s sys 0.001s
I20260926 01:57:14.904035 12433 ts_tablet_manager.cc:1434] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
I20260926 01:57:14.906034 12435 ts_tablet_manager.cc:1929] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:14.906441 12435 log.cc:1199] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:14.907334 12435 ts_tablet_manager.cc:1950] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:14.909714 12150 tablet_service.cc:1511] Processing CreateTablet for tablet 6ac6c587687c4d868586da53310dfa5b (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.910211 11654 catalog_manager.cc:5063] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): tablet 5ab9e06a641740c7bbb1afbe0f4943a0 (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:14.910950 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ac6c587687c4d868586da53310dfa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.911356 12433 tablet_bootstrap.cc:492] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:14.916679 12437 tablet_bootstrap.cc:492] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:14.917294 12437 ts_tablet_manager.cc:1403] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.008s
I20260926 01:57:14.920686 12437 raft_consensus.cc:359] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.926038 12005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ac6c587687c4d868586da53310dfa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.926200 12437 raft_consensus.cc:385] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.926678 12437 raft_consensus.cc:740] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.927542 12437 consensus_queue.cc:260] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.937951 12437 ts_tablet_manager.cc:1434] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.020s user 0.003s sys 0.003s
I20260926 01:57:14.939136 12437 tablet_bootstrap.cc:492] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:14.926075 12438 ts_tablet_manager.cc:1929] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:14.940990 12438 log.cc:1199] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/c34c5cb77d164ddeb904c3b54aba569e
I20260926 01:57:14.942008 12438 ts_tablet_manager.cc:1950] T c34c5cb77d164ddeb904c3b54aba569e P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting consensus metadata
I20260926 01:57:14.946259 12437 tablet_bootstrap.cc:654] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:14.956697 12433 tablet_bootstrap.cc:654] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
W20260926 01:57:14.974248 12079 tablet.cc:2433] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:57:14.976588 12295 tablet_service.cc:1511] Processing CreateTablet for tablet 6ac6c587687c4d868586da53310dfa5b (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:14.977087 12431 raft_consensus.cc:359] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:14.977820 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ac6c587687c4d868586da53310dfa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:14.977793 12431 raft_consensus.cc:385] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:14.978158 12431 raft_consensus.cc:740] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:14.983682 11657 catalog_manager.cc:5063] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): tablet c34c5cb77d164ddeb904c3b54aba569e (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:14.984104 12438 tablet_replica.cc:333] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: stopping tablet replica
I20260926 01:57:14.984912 12438 raft_consensus.cc:2243] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:14.985452 12438 raft_consensus.cc:2272] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260926 01:57:14.985756 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet 5ab9e06a641740c7bbb1afbe0f4943a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ab9e06a641740c7bbb1afbe0f4943a0 already in progress: creating tablet
I20260926 01:57:14.987864 12438 ts_tablet_manager.cc:1916] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:14.982446 12431 consensus_queue.cc:260] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.007746 12431 ts_tablet_manager.cc:1434] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.104s user 0.007s sys 0.000s
I20260926 01:57:15.008853 12431 tablet_bootstrap.cc:492] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:15.009689 12433 tablet_bootstrap.cc:492] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.010829 11664 master_service.cc:960] ReplaceTablet: received request to replace tablet 68a9475fbf434bd688e2860929764a6f from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:15.010213 12433 ts_tablet_manager.cc:1403] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.005s
I20260926 01:57:15.020207 11664 master_service.cc:960] ReplaceTablet: received request to replace tablet 1d661349b7d54176ad406fd43872b8dd from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:15.022157 12433 raft_consensus.cc:359] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.033367 12433 raft_consensus.cc:385] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.033670 12431 tablet_bootstrap.cc:654] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.033748 12433 raft_consensus.cc:740] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
W20260926 01:57:15.035121 12371 tablet.cc:2433] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:57:15.043430 12433 consensus_queue.cc:260] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.053665 11664 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Sending DeleteTablet for 3 replicas of tablet 1d661349b7d54176ad406fd43872b8dd
I20260926 01:57:15.056502 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 10a5289af7464d959096cdaaca6d12e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b554bcce374d4e5fa02957cd8aaebde8) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:15.057361 12436 tablet_replica.cc:333] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
I20260926 01:57:15.058085 12436 raft_consensus.cc:2243] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.058297 12433 ts_tablet_manager.cc:1434] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260926 01:57:15.058642 12436 raft_consensus.cc:2272] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.061013 12436 ts_tablet_manager.cc:1916] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.062539 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet 10a5289af7464d959096cdaaca6d12e0 with delete_type TABLET_DATA_DELETED (replaced by tablet b554bcce374d4e5fa02957cd8aaebde8) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:15.066856 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d661349b7d54176ad406fd43872b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 2ea4233817144476b1397f95eb68c5da) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:15.066783 12151 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d661349b7d54176ad406fd43872b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 2ea4233817144476b1397f95eb68c5da) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:15.067575 12435 tablet_replica.cc:333] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:15.067737 12433 tablet_bootstrap.cc:492] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:15.068306 12435 raft_consensus.cc:2243] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.068821 12435 raft_consensus.cc:2272] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.069577 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d661349b7d54176ad406fd43872b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 2ea4233817144476b1397f95eb68c5da) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:15.071810 12435 ts_tablet_manager.cc:1916] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.082612 11664 catalog_manager.cc:6934] ReplaceTablet: tablet 1d661349b7d54176ad406fd43872b8dd deleted and replaced by tablet 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:15.085306 12437 tablet_bootstrap.cc:492] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:15.085875 12437 ts_tablet_manager.cc:1403] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.147s user 0.013s sys 0.042s
I20260926 01:57:15.088822 12437 raft_consensus.cc:359] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.089491 12433 tablet_bootstrap.cc:654] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.089576 12437 raft_consensus.cc:385] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.091571 12437 raft_consensus.cc:740] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.091768 12436 ts_tablet_manager.cc:1929] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.092187 12436 log.cc:1199] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/10a5289af7464d959096cdaaca6d12e0
I20260926 01:57:15.092908 12436 ts_tablet_manager.cc:1950] T 10a5289af7464d959096cdaaca6d12e0 P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
I20260926 01:57:15.092850 12437 consensus_queue.cc:260] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.098567 12438 ts_tablet_manager.cc:1929] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.099027 12438 log.cc:1199] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/68a9475fbf434bd688e2860929764a6f
I20260926 01:57:15.099323 11654 catalog_manager.cc:5063] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): tablet 10a5289af7464d959096cdaaca6d12e0 (table default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]) successfully deleted
I20260926 01:57:15.099757 12438 ts_tablet_manager.cc:1950] T 68a9475fbf434bd688e2860929764a6f P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting consensus metadata
I20260926 01:57:15.102784 11657 catalog_manager.cc:5063] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): tablet 68a9475fbf434bd688e2860929764a6f (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]) successfully deleted
W20260926 01:57:15.103595 11654 catalog_manager.cc:5043] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): delete failed for tablet 1d661349b7d54176ad406fd43872b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d661349b7d54176ad406fd43872b8dd already in progress: creating tablet
I20260926 01:57:15.109591 12438 tablet_replica.cc:333] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: stopping tablet replica
I20260926 01:57:15.111160 12438 raft_consensus.cc:2243] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.111683 12438 raft_consensus.cc:2272] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.116909 12433 tablet_bootstrap.cc:492] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.117377 12433 ts_tablet_manager.cc:1403] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20260926 01:57:15.120556 12433 raft_consensus.cc:359] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.121265 12433 raft_consensus.cc:385] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.121558 12433 raft_consensus.cc:740] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.124056 12438 ts_tablet_manager.cc:1916] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.129544 12437 ts_tablet_manager.cc:1434] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.043s user 0.005s sys 0.001s
I20260926 01:57:15.138213 12437 tablet_bootstrap.cc:492] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:15.135076 12433 consensus_queue.cc:260] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.145591 12437 tablet_bootstrap.cc:654] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.146792 12435 ts_tablet_manager.cc:1929] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.147369 12435 log.cc:1199] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/1d661349b7d54176ad406fd43872b8dd
I20260926 01:57:15.147822 12433 ts_tablet_manager.cc:1434] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
W20260926 01:57:15.149072 11664 catalog_manager.cc:5671] Ignoring report from unknown tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:15.150640 12435 ts_tablet_manager.cc:1950] T 1d661349b7d54176ad406fd43872b8dd P 1b877862e4a543229352921130f57068: Deleting consensus metadata
W20260926 01:57:15.154271 12223 tablet.cc:2433] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 01:57:15.154974 12433 tablet_bootstrap.cc:492] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:15.156569 11654 catalog_manager.cc:5063] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): tablet 1d661349b7d54176ad406fd43872b8dd (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]) successfully deleted
I20260926 01:57:15.155529 12431 tablet_bootstrap.cc:492] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:15.158120 12431 ts_tablet_manager.cc:1403] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.149s user 0.010s sys 0.003s
W20260926 01:57:15.162147 11664 catalog_manager.cc:5671] Ignoring report from unknown tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:15.173691 12431 raft_consensus.cc:359] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.180706 12005 tablet_service.cc:1511] Processing CreateTablet for tablet 2ea4233817144476b1397f95eb68c5da (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.181977 12005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea4233817144476b1397f95eb68c5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:15.190645 12431 raft_consensus.cc:385] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.191160 12431 raft_consensus.cc:740] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.201068 12433 tablet_bootstrap.cc:654] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.202077 12150 tablet_service.cc:1511] Processing CreateTablet for tablet 2ea4233817144476b1397f95eb68c5da (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.202828 12295 tablet_service.cc:1511] Processing CreateTablet for tablet 2ea4233817144476b1397f95eb68c5da (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.203549 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea4233817144476b1397f95eb68c5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:15.203974 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea4233817144476b1397f95eb68c5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:15.203559 12431 consensus_queue.cc:260] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:15.216645 11663 catalog_manager.cc:5671] Ignoring report from unknown tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:15.220873 12431 ts_tablet_manager.cc:1434] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20260926 01:57:15.224696 12437 tablet_bootstrap.cc:492] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:15.225410 11663 master_service.cc:960] ReplaceTablet: received request to replace tablet 2ea4233817144476b1397f95eb68c5da from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:15.237401 12438 ts_tablet_manager.cc:1929] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.240139 12438 log.cc:1199] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/10a5289af7464d959096cdaaca6d12e0
I20260926 01:57:15.241092 12438 ts_tablet_manager.cc:1950] T 10a5289af7464d959096cdaaca6d12e0 P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting consensus metadata
I20260926 01:57:15.242070 12431 tablet_bootstrap.cc:492] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:15.235981 12446 raft_consensus.cc:493] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:15.242858 12446 raft_consensus.cc:515] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:15.251587 11664 catalog_manager.cc:5671] Ignoring report from unknown tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:15.254624 11663 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Sending DeleteTablet for 3 replicas of tablet 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:15.256383 11663 catalog_manager.cc:6934] ReplaceTablet: tablet 2ea4233817144476b1397f95eb68c5da deleted and replaced by tablet 6ab1ce3df47c44c9bb38768bc4c44d7f
I20260926 01:57:15.261897 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ea4233817144476b1397f95eb68c5da with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab1ce3df47c44c9bb38768bc4c44d7f) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:15.263608 12446 leader_election.cc:290] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:15.269667 11654 catalog_manager.cc:5043] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): delete failed for tablet 2ea4233817144476b1397f95eb68c5da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ea4233817144476b1397f95eb68c5da already in progress: creating tablet
I20260926 01:57:15.274012 12437 ts_tablet_manager.cc:1403] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.136s user 0.008s sys 0.007s
I20260926 01:57:15.276932 12437 raft_consensus.cc:359] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.258215 12296 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ea4233817144476b1397f95eb68c5da with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab1ce3df47c44c9bb38768bc4c44d7f) from {username='slave'} at 127.0.0.1:59244
W20260926 01:57:15.284608 11654 catalog_manager.cc:5043] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): delete failed for tablet 2ea4233817144476b1397f95eb68c5da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ea4233817144476b1397f95eb68c5da already in progress: creating tablet
I20260926 01:57:15.302030 12150 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.302325 12437 raft_consensus.cc:385] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.302670 12437 raft_consensus.cc:740] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.303294 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:15.303792 12433 tablet_bootstrap.cc:492] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.304328 12433 ts_tablet_manager.cc:1403] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.150s user 0.017s sys 0.020s
I20260926 01:57:15.304198 12295 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.305341 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:15.307261 12433 raft_consensus.cc:359] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.307217 12437 consensus_queue.cc:260] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.308056 12433 raft_consensus.cc:385] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.308404 12433 raft_consensus.cc:740] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.310501 12433 consensus_queue.cc:260] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.325052 12437 ts_tablet_manager.cc:1434] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
01:57:15.326 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_7440abc68fd740d49170b27f2cb1c731 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
W20260926 01:57:15.330266 11931 hms_client.cc:335] Time spent create HMS table: real 1.187s user 0.000s sys 0.000s
W20260926 01:57:15.331280 11668 catalog_manager.cc:2209] Already present: failed to create HMS catalog entry for table [id=8a31d897c4cb42a5a46c03fd2152e795]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_7440abc68fd740d49170b27f2cb1c731 already exists)
I20260926 01:57:15.332233 11668 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:43138 (request call id 24) took 1864 ms. Trace:
I20260926 01:57:15.332573 11668 rpcz_store.cc:276] 0926 01:57:13.468082 (+ 0us) service_pool.cc:167] Inserting onto call queue
0926 01:57:13.468238 (+ 156us) service_pool.cc:224] Handling call
0926 01:57:13.542779 (+ 74541us) catalog_manager.cc:2140] Acquired catalog manager lock
0926 01:57:13.544782 (+ 2003us) catalog_manager.cc:2186] Created new table and tablet info
0926 01:57:15.332153 (+1787371us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":588773,"Hive Metastore.run_cpu_time_us":2234,"Hive Metastore.run_wall_time_us":1187114}
I20260926 01:57:15.338222 11663 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43138:
name: "default.Table_9aa9a3fdad284796b3f433da53236871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:15.341789 12437 tablet_bootstrap.cc:492] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: Bootstrap starting.
F20260926 01:57:15.346518 11933 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1790387833, eventType=CREATE_TABLE, dbName=default, tableName=table_7440abc68fd740d49170b27f2cb1c731, message={"server":"","servicePrincipal":"","db":"default","table":"table_7440abc68fd740d49170b27f2cb1c731","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7440abc68fd740d49170b27f2cb1c731\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790387833},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/warehouse/table_7440abc68fd740d49170b27f2cb1c731\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7440abc68fd740d49170b27f2cb1c731\",\"kudu.table_id\":\"a49a2e5756be4c1384ad85aba0834a58\",\"transient_lastDdlTime\":\"1790387833\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"29d8d2959c5241009da641125beda276\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1790387833,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1790387833, eventType=CREATE_TABLE, dbName=default, tableName=table_9ecb0b51e0e947d4bc9d0fb60139994f, message={"server":"","servicePrincipal":"","db":"default","table":"table_9ecb0b51e0e947d4bc9d0fb60139994f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9ecb0b51e0e947d4bc9d0fb60139994f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790387833},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/warehouse/table_9ecb0b51e0e947d4bc9d0fb60139994f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9ecb0b51e0e947d4bc9d0fb60139994f\",\"kudu.table_id\":\"c3c32452d955463485f7f60723c019ec\",\"transient_lastDdlTime\":\"1790387833\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"29d8d2959c5241009da641125beda276\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1790387833,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260926 01:57:15.353277 12437 tablet_bootstrap.cc:654] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.361126 11657 catalog_manager.cc:5063] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): tablet 10a5289af7464d959096cdaaca6d12e0 (table default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]) successfully deleted
W20260926 01:57:15.362013 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet 1d661349b7d54176ad406fd43872b8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d661349b7d54176ad406fd43872b8dd already in progress: creating tablet
I20260926 01:57:15.376971 12431 tablet_bootstrap.cc:654] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.376708 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b554bcce374d4e5fa02957cd8aaebde8" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3" is_pre_election: true
I20260926 01:57:15.379321 11668 master_service.cc:960] ReplaceTablet: received request to replace tablet 1d661349b7d54176ad406fd43872b8dd from {username='slave'} at 127.0.0.1:43138
W20260926 01:57:15.379150 12104 leader_election.cc:343] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:57:15.393296 12433 ts_tablet_manager.cc:1434] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.089s user 0.005s sys 0.004s
I20260926 01:57:15.383221 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ea4233817144476b1397f95eb68c5da with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab1ce3df47c44c9bb38768bc4c44d7f) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:15.384131 12004 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.395794 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:15.397033 12433 tablet_bootstrap.cc:492] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:15.404284 12431 tablet_bootstrap.cc:492] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:15.404832 12431 ts_tablet_manager.cc:1403] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.163s user 0.012s sys 0.000s
I20260926 01:57:15.407968 12431 raft_consensus.cc:359] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.408684 12431 raft_consensus.cc:385] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.408970 12431 raft_consensus.cc:740] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
@ 0x7f74927eb920 google::LogMessage::Flush() at ??:0
I20260926 01:57:15.413219 12433 tablet_bootstrap.cc:654] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.413393 12431 consensus_queue.cc:260] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.416195 12431 ts_tablet_manager.cc:1434] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260926 01:57:15.416298 12437 tablet_bootstrap.cc:492] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:15.416805 12437 ts_tablet_manager.cc:1403] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.075s user 0.020s sys 0.024s
I20260926 01:57:15.417176 12431 tablet_bootstrap.cc:492] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:15.419561 12449 raft_consensus.cc:493] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:15.419677 12437 raft_consensus.cc:359] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.420332 12437 raft_consensus.cc:385] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.420607 12437 raft_consensus.cc:740] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.420428 12449 raft_consensus.cc:515] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.421277 12437 consensus_queue.cc:260] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:15.422823 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet 2ea4233817144476b1397f95eb68c5da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ea4233817144476b1397f95eb68c5da already in progress: creating tablet
I20260926 01:57:15.425654 12005 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d661349b7d54176ad406fd43872b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 2ea4233817144476b1397f95eb68c5da) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:15.426640 12438 tablet_replica.cc:333] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: stopping tablet replica
I20260926 01:57:15.427418 12438 raft_consensus.cc:2243] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.427940 12438 raft_consensus.cc:2272] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.430982 12437 ts_tablet_manager.cc:1434] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260926 01:57:15.432030 12437 tablet_bootstrap.cc:492] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:15.437605 12431 tablet_bootstrap.cc:654] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
@ 0x7f74927eb68f google::LogMessage::~LogMessage() at ??:0
I20260926 01:57:15.444355 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ea4233817144476b1397f95eb68c5da with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab1ce3df47c44c9bb38768bc4c44d7f) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:15.445258 12435 tablet_replica.cc:333] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:15.446063 12435 raft_consensus.cc:2243] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.446601 12435 raft_consensus.cc:2272] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.447953 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d661349b7d54176ad406fd43872b8dd with delete_type TABLET_DATA_DELETED (replaced by tablet 2ea4233817144476b1397f95eb68c5da) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:15.448639 12437 tablet_bootstrap.cc:654] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.448776 12436 tablet_replica.cc:333] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
I20260926 01:57:15.449538 12436 raft_consensus.cc:2243] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.450023 12436 raft_consensus.cc:2272] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.452342 12436 ts_tablet_manager.cc:1916] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.452572 12435 ts_tablet_manager.cc:1916] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.471303 12438 ts_tablet_manager.cc:1916] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.478880 12431 tablet_bootstrap.cc:492] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:15.479389 12431 ts_tablet_manager.cc:1403] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.004s
I20260926 01:57:15.482214 12449 leader_election.cc:290] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:15.482518 11664 master_service.cc:960] ReplaceTablet: received request to replace tablet 5ab9e06a641740c7bbb1afbe0f4943a0 from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:15.488520 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b554bcce374d4e5fa02957cd8aaebde8" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" is_pre_election: true
W20260926 01:57:15.490770 12107 leader_election.cc:343] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:57:15.491268 12107 leader_election.cc:304] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b877862e4a543229352921130f57068; no voters: 495ab022dafc493395ae0e21d9a77ee3, 5d58342c1ed14ef9a6815b9b5a219b56
I20260926 01:57:15.492691 12446 raft_consensus.cc:2749] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:15.485834 12431 raft_consensus.cc:359] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.496981 12431 raft_consensus.cc:385] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.497272 12431 raft_consensus.cc:740] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.500429 12431 consensus_queue.cc:260] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
@ 0x7f749d97e35c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260926 01:57:15.509135 12431 ts_tablet_manager.cc:1434] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.029s user 0.003s sys 0.004s
I20260926 01:57:15.514847 12437 tablet_bootstrap.cc:492] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:15.515326 12437 ts_tablet_manager.cc:1403] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.084s user 0.015s sys 0.007s
I20260926 01:57:15.518157 12437 raft_consensus.cc:359] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.518812 12437 raft_consensus.cc:385] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.519074 12437 raft_consensus.cc:740] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.519719 12437 consensus_queue.cc:260] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.521605 12437 ts_tablet_manager.cc:1434] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260926 01:57:15.539112 12431 tablet_bootstrap.cc:492] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:15.539992 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ab9e06a641740c7bbb1afbe0f4943a0" candidate_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3" is_pre_election: true
I20260926 01:57:15.540830 12316 raft_consensus.cc:2468] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d58342c1ed14ef9a6815b9b5a219b56 in term 0.
I20260926 01:57:15.542553 11959 leader_election.cc:304] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495ab022dafc493395ae0e21d9a77ee3, 5d58342c1ed14ef9a6815b9b5a219b56; no voters:
I20260926 01:57:15.496542 12436 ts_tablet_manager.cc:1929] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.543601 12449 raft_consensus.cc:2804] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:57:15.544001 12449 raft_consensus.cc:493] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:57:15.544041 12436 log.cc:1199] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/1d661349b7d54176ad406fd43872b8dd
I20260926 01:57:15.544353 12449 raft_consensus.cc:3060] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:15.544839 12436 ts_tablet_manager.cc:1950] T 1d661349b7d54176ad406fd43872b8dd P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
@ 0x7f749d97db1c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260926 01:57:15.558264 12449 raft_consensus.cc:515] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.560724 12449 leader_election.cc:290] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:15.562862 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ab9e06a641740c7bbb1afbe0f4943a0" candidate_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3"
W20260926 01:57:15.565158 11959 leader_election.cc:343] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:15.563478 12316 raft_consensus.cc:3060] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:57:15.571002 11959 leader_election.cc:343] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:15.571835 11654 catalog_manager.cc:5063] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): tablet 1d661349b7d54176ad406fd43872b8dd (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]) successfully deleted
I20260926 01:57:15.576776 12316 raft_consensus.cc:2468] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d58342c1ed14ef9a6815b9b5a219b56 in term 1.
I20260926 01:57:15.578214 11959 leader_election.cc:304] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 495ab022dafc493395ae0e21d9a77ee3, 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068
I20260926 01:57:15.582144 12433 tablet_bootstrap.cc:492] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.582696 12433 ts_tablet_manager.cc:1403] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.007s
I20260926 01:57:15.588196 12431 tablet_bootstrap.cc:654] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.588748 12449 raft_consensus.cc:2804] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:57:15.594893 12438 ts_tablet_manager.cc:1929] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.595378 12438 log.cc:1199] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/1d661349b7d54176ad406fd43872b8dd
I20260926 01:57:15.597012 12435 ts_tablet_manager.cc:1929] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.597520 12435 log.cc:1199] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/2ea4233817144476b1397f95eb68c5da
I20260926 01:57:15.598163 12435 ts_tablet_manager.cc:1950] T 2ea4233817144476b1397f95eb68c5da P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:15.601200 11654 catalog_manager.cc:5063] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): tablet 2ea4233817144476b1397f95eb68c5da (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]) successfully deleted
I20260926 01:57:15.601737 12449 raft_consensus.cc:697] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 LEADER]: Becoming Leader. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Running, Role: LEADER
I20260926 01:57:15.602809 12449 consensus_queue.cc:237] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.605921 12433 raft_consensus.cc:359] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.606665 12433 raft_consensus.cc:385] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.606962 12433 raft_consensus.cc:740] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.607760 12433 consensus_queue.cc:260] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
@ 0x7f749d981b92 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260926 01:57:15.620359 12438 ts_tablet_manager.cc:1950] T 1d661349b7d54176ad406fd43872b8dd P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting consensus metadata
I20260926 01:57:15.622119 11657 catalog_manager.cc:5063] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): tablet 1d661349b7d54176ad406fd43872b8dd (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]) successfully deleted
I20260926 01:57:15.631300 12431 tablet_bootstrap.cc:492] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:15.631791 12431 ts_tablet_manager.cc:1403] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.006s
I20260926 01:57:15.634826 12431 raft_consensus.cc:359] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.635514 12431 raft_consensus.cc:385] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.635751 12431 raft_consensus.cc:740] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
@ 0x7f749d981b4a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
W20260926 01:57:15.645104 11668 catalog_manager.cc:5671] Ignoring report from unknown tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:15.649158 12431 consensus_queue.cc:260] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.656873 11668 master_service.cc:960] ReplaceTablet: received request to replace tablet 6ac6c587687c4d868586da53310dfa5b from {username='slave'} at 127.0.0.1:43138
@ 0x7f749d981ada std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260926 01:57:15.659988 12431 ts_tablet_manager.cc:1434] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260926 01:57:15.674880 12431 tablet_bootstrap.cc:492] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
@ 0x7f749d981aa2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260926 01:57:15.694962 11661 master_service.cc:960] ReplaceTablet: received request to replace tablet 2ea4233817144476b1397f95eb68c5da from {username='slave'} at 127.0.0.1:43138
I20260926 01:57:15.695820 12431 tablet_bootstrap.cc:654] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.697952 12433 ts_tablet_manager.cc:1434] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.115s user 0.000s sys 0.013s
I20260926 01:57:15.699133 11668 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Sending DeleteTablet for 3 replicas of tablet 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:15.701126 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ac6c587687c4d868586da53310dfa5b with delete_type TABLET_DATA_DELETED (replaced by tablet dde7568a319f44dbb455621070675a23) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:15.704026 12004 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ac6c587687c4d868586da53310dfa5b with delete_type TABLET_DATA_DELETED (replaced by tablet dde7568a319f44dbb455621070675a23) from {username='slave'} at 127.0.0.1:49028
I20260926 01:57:15.704914 12436 tablet_replica.cc:333] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
I20260926 01:57:15.705771 12436 raft_consensus.cc:2243] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.706297 12436 raft_consensus.cc:2272] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.709916 12436 ts_tablet_manager.cc:1916] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f749d980d9e std::__1::__function::__func<>::operator()() at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
I20260926 01:57:15.713841 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ac6c587687c4d868586da53310dfa5b with delete_type TABLET_DATA_DELETED (replaced by tablet dde7568a319f44dbb455621070675a23) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:15.714005 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ac6c587687c4d868586da53310dfa5b with delete_type TABLET_DATA_DELETED (replaced by tablet dde7568a319f44dbb455621070675a23) from {username='slave'} at 127.0.0.1:53088
I20260926 01:57:15.714912 12435 tablet_replica.cc:333] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:15.715816 12435 raft_consensus.cc:2243] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.715940 11668 catalog_manager.cc:6934] ReplaceTablet: tablet 6ac6c587687c4d868586da53310dfa5b deleted and replaced by tablet dde7568a319f44dbb455621070675a23
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
I20260926 01:57:15.716388 12435 raft_consensus.cc:2272] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260926 01:57:15.716573 11657 catalog_manager.cc:5043] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): delete failed for tablet 6ac6c587687c4d868586da53310dfa5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ac6c587687c4d868586da53310dfa5b already in progress: creating tablet
I20260926 01:57:15.718954 12435 ts_tablet_manager.cc:1916] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.719046 12433 tablet_bootstrap.cc:492] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:15.731307 12433 tablet_bootstrap.cc:654] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.748992 12431 tablet_bootstrap.cc:492] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:15.749485 12431 ts_tablet_manager.cc:1403] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.000s
I20260926 01:57:15.752187 12431 raft_consensus.cc:359] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.752879 12431 raft_consensus.cc:385] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.753139 12431 raft_consensus.cc:740] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.753715 12435 ts_tablet_manager.cc:1929] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.754078 12435 log.cc:1199] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:15.754827 12435 ts_tablet_manager.cc:1950] T 6ac6c587687c4d868586da53310dfa5b P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:15.757638 11654 catalog_manager.cc:5063] TS 1b877862e4a543229352921130f57068 (127.8.76.130:34281): tablet 6ac6c587687c4d868586da53310dfa5b (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:15.758556 12431 consensus_queue.cc:260] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
@ 0x7f74949e181b kudu::Thread::SuperviseThread() at ??:0
I20260926 01:57:15.760658 12431 ts_tablet_manager.cc:1434] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
@ 0x4569a0 __tsan_thread_start_func at ??:?
I20260926 01:57:15.761550 12431 tablet_bootstrap.cc:492] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:15.771186 12004 tablet_service.cc:1511] Processing CreateTablet for tablet dde7568a319f44dbb455621070675a23 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.771883 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ea4233817144476b1397f95eb68c5da with delete_type TABLET_DATA_DELETED (replaced by tablet 6ab1ce3df47c44c9bb38768bc4c44d7f) from {username='slave'} at 127.0.0.1:59244
I20260926 01:57:15.772526 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde7568a319f44dbb455621070675a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:15.777828 12436 ts_tablet_manager.cc:1929] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
@ 0x7f74961426db start_thread at ??:0
I20260926 01:57:15.778249 12436 log.cc:1199] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:15.779008 12436 ts_tablet_manager.cc:1950] T 6ac6c587687c4d868586da53310dfa5b P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
I20260926 01:57:15.782009 11654 catalog_manager.cc:5063] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): tablet 6ac6c587687c4d868586da53310dfa5b (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:15.786707 12150 tablet_service.cc:1511] Processing CreateTablet for tablet dde7568a319f44dbb455621070675a23 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:15.787949 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde7568a319f44dbb455621070675a23. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 01:57:15.792158 11654 catalog_manager.cc:5030] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): delete failed for tablet 6ac6c587687c4d868586da53310dfa5b because the tablet was not found. No further retry: Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:15.793274 12436 tablet_replica.cc:333] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
@ 0x7f749148d61f clone at ??:0
I20260926 01:57:15.794067 12436 raft_consensus.cc:2243] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:15.797364 12431 tablet_bootstrap.cc:654] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.800223 12436 raft_consensus.cc:2272] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:15.803447 12436 ts_tablet_manager.cc:1916] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:15.811611 12446 raft_consensus.cc:493] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:15.812150 12446 raft_consensus.cc:515] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.817252 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab1ce3df47c44c9bb38768bc4c44d7f" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" is_pre_election: true
I20260926 01:57:15.818686 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab1ce3df47c44c9bb38768bc4c44d7f" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3" is_pre_election: true
W20260926 01:57:15.820703 12104 leader_election.cc:343] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Illegal state: must be running to vote when last-logged opid is not known
W20260926 01:57:15.822028 12107 leader_election.cc:343] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): Illegal state: must be running to vote when last-logged opid is not known
I20260926 01:57:15.822494 12107 leader_election.cc:304] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b877862e4a543229352921130f57068; no voters: 495ab022dafc493395ae0e21d9a77ee3, 5d58342c1ed14ef9a6815b9b5a219b56
I20260926 01:57:15.814368 12446 leader_election.cc:290] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:15.824134 12446 raft_consensus.cc:2749] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:15.827422 12431 tablet_bootstrap.cc:492] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:15.827503 12437 tablet_bootstrap.cc:492] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:15.827873 12431 ts_tablet_manager.cc:1403] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.004s
I20260926 01:57:15.833590 12437 tablet_bootstrap.cc:654] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.834386 12433 tablet_bootstrap.cc:492] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.835052 12433 ts_tablet_manager.cc:1403] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.116s user 0.012s sys 0.000s
I20260926 01:57:15.837706 12433 raft_consensus.cc:359] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.838354 12433 raft_consensus.cc:385] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.838658 12433 raft_consensus.cc:740] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.839360 12433 consensus_queue.cc:260] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.841780 12433 ts_tablet_manager.cc:1434] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 01:57:15.830435 12431 raft_consensus.cc:359] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.842912 12431 raft_consensus.cc:385] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.843195 12431 raft_consensus.cc:740] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.843948 12431 consensus_queue.cc:260] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.845943 12431 ts_tablet_manager.cc:1434] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20260926 01:57:15.855497 12433 tablet_bootstrap.cc:492] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:15.874192 12437 tablet_bootstrap.cc:492] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:15.875067 12433 tablet_bootstrap.cc:654] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.875164 12437 ts_tablet_manager.cc:1403] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.048s user 0.027s sys 0.014s
I20260926 01:57:15.877849 12437 raft_consensus.cc:359] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.878563 12437 raft_consensus.cc:385] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.878818 12437 raft_consensus.cc:740] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.879590 12437 consensus_queue.cc:260] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.882042 12437 ts_tablet_manager.cc:1434] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 01:57:15.905864 12433 tablet_bootstrap.cc:492] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.906287 12433 ts_tablet_manager.cc:1403] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.002s
I20260926 01:57:15.909032 12433 raft_consensus.cc:359] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.909682 12433 raft_consensus.cc:385] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.909920 12433 raft_consensus.cc:740] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.910569 12433 consensus_queue.cc:260] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.912679 12433 ts_tablet_manager.cc:1434] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 01:57:15.913539 12433 tablet_bootstrap.cc:492] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:15.921379 12433 tablet_bootstrap.cc:654] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.942979 12433 tablet_bootstrap.cc:492] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.944190 12433 ts_tablet_manager.cc:1403] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.003s
I20260926 01:57:15.946820 12433 raft_consensus.cc:359] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.947449 12433 raft_consensus.cc:385] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.947685 12433 raft_consensus.cc:740] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.948314 12433 consensus_queue.cc:260] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.950498 12433 ts_tablet_manager.cc:1434] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 01:57:15.951310 12433 tablet_bootstrap.cc:492] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:15.957628 12433 tablet_bootstrap.cc:654] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:15.965973 12436 ts_tablet_manager.cc:1929] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260926 01:57:15.966406 12436 log.cc:1199] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/2ea4233817144476b1397f95eb68c5da
I20260926 01:57:15.967104 12436 ts_tablet_manager.cc:1950] T 2ea4233817144476b1397f95eb68c5da P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
I20260926 01:57:15.967783 12433 tablet_bootstrap.cc:492] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:15.968195 12433 ts_tablet_manager.cc:1403] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260926 01:57:15.977295 12433 raft_consensus.cc:359] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.977977 12433 raft_consensus.cc:385] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:15.978226 12433 raft_consensus.cc:740] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:15.978932 12433 consensus_queue.cc:260] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:15.980751 12433 ts_tablet_manager.cc:1434] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260926 01:57:16.061353 12443 raft_consensus.cc:493] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:16.061918 12443 raft_consensus.cc:515] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.075126 12443 leader_election.cc:290] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:16.088280 12171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b554bcce374d4e5fa02957cd8aaebde8" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b877862e4a543229352921130f57068" is_pre_election: true
I20260926 01:57:16.089172 12171 raft_consensus.cc:2468] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 0.
I20260926 01:57:16.091117 12249 leader_election.cc:304] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3; no voters:
I20260926 01:57:16.092304 12443 raft_consensus.cc:2804] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:57:16.092818 12443 raft_consensus.cc:493] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:57:16.093233 12443 raft_consensus.cc:3060] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:16.095005 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b554bcce374d4e5fa02957cd8aaebde8" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" is_pre_election: true
I20260926 01:57:16.095677 12026 raft_consensus.cc:2468] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 0.
I20260926 01:57:16.100487 12443 raft_consensus.cc:515] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.103341 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b554bcce374d4e5fa02957cd8aaebde8" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56"
I20260926 01:57:16.103674 12171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b554bcce374d4e5fa02957cd8aaebde8" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b877862e4a543229352921130f57068"
I20260926 01:57:16.103922 12026 raft_consensus.cc:3060] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:16.104259 12171 raft_consensus.cc:3060] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Advancing to term 1
W20260926 01:57:16.108445 12421 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:57:16.109304 12421 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:57:16.110065 12421 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260926 01:57:16.111456 12171 raft_consensus.cc:2468] T b554bcce374d4e5fa02957cd8aaebde8 P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 1.
I20260926 01:57:16.112673 12026 raft_consensus.cc:2468] T b554bcce374d4e5fa02957cd8aaebde8 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 1.
I20260926 01:57:16.112841 12249 leader_election.cc:304] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3; no voters:
I20260926 01:57:16.114528 12443 leader_election.cc:290] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 election: Requested vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:16.115160 12443 raft_consensus.cc:2804] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:57:16.115803 12443 raft_consensus.cc:697] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Running, Role: LEADER
I20260926 01:57:16.116761 12443 consensus_queue.cc:237] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:16.155062 12421 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260926 01:57:16.155383 12421 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260926 01:57:16.155655 12421 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:57:16.155929 12421 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:57:16.156190 12421 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:57:16.156490 12421 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260926 01:57:16.158972 12249 connection.cc:573] server connection from 127.0.0.1:59244 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:16.158967 12107 connection.cc:573] server connection from 127.0.0.1:53088 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:16.159134 11961 connection.cc:573] server connection from 127.0.0.1:49028 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:16.160238 12374 connection.cc:573] client connection to 127.8.76.190:41989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:16.162532 12251 connection.cc:573] client connection to 127.8.76.190:41989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:16.164552 12374 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.76.190:41989)
W20260926 01:57:16.166013 12374 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.190:41989)
W20260926 01:57:16.168694 12374 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.76.190:41989)
W20260926 01:57:16.168879 12106 connection.cc:573] client connection to 127.8.76.190:41989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:16.169762 12374 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.76.190:41989)
W20260926 01:57:16.171515 11961 connection.cc:573] client connection to 127.8.76.190:41989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:16.178133 12441 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:16.173709 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.8.76.190:41989 (local address 127.8.76.129:50011)
0926 01:57:16.174776 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0926 01:57:16.174823 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0926 01:57:16.175078 (+ 255us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0926 01:57:16.176940 (+ 1862us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":383}
W20260926 01:57:16.181603 12507 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:16.167153 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.8.76.190:41989 (local address 127.8.76.131:49161)
0926 01:57:16.170592 (+ 3439us) negotiation.cc:107] Waiting for socket to connect
0926 01:57:16.170648 (+ 56us) client_negotiation.cc:175] Beginning negotiation
0926 01:57:16.170966 (+ 318us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0926 01:57:16.180315 (+ 9349us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2745}
I20260926 01:57:16.184942 12482 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:16.185456 12482 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.187554 12482 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:16.190409 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:57:16.191622 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:16.192145 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:16.193286 12482 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:57:16.207094 12510 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:16.172615 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.8.76.190:41989 (local address 127.0.0.1:59840)
0926 01:57:16.196454 (+ 23839us) negotiation.cc:107] Waiting for socket to connect
0926 01:57:16.196712 (+ 258us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":23204,"threads_started":1}
I20260926 01:57:16.225584 12421 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43863
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.188:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.76.188
--webserver_port=46041
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:57:16.227360 12421 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:57:16.229238 12421 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:57:16.249224 12513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:57:16.273084 12518 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9760316a8f74e829b4c7884b875b70e)
I20260926 01:57:16.273613 12518 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:16.288327 12518 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:16.326668 11728 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:16.327409 11730 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
I20260926 01:57:16.327853 11730 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:16.329162 12518 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260926 01:57:16.502326 12443 raft_consensus.cc:493] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:16.503091 12443 raft_consensus.cc:515] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.508339 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab1ce3df47c44c9bb38768bc4c44d7f" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" is_pre_election: true
I20260926 01:57:16.509147 12026 raft_consensus.cc:2468] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 0.
I20260926 01:57:16.510617 12443 leader_election.cc:290] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:16.512159 12171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab1ce3df47c44c9bb38768bc4c44d7f" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b877862e4a543229352921130f57068" is_pre_election: true
I20260926 01:57:16.512933 12171 raft_consensus.cc:2468] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 0.
I20260926 01:57:16.514977 12252 leader_election.cc:304] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495ab022dafc493395ae0e21d9a77ee3, 5d58342c1ed14ef9a6815b9b5a219b56; no voters:
I20260926 01:57:16.518329 12443 raft_consensus.cc:2804] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:57:16.518836 12443 raft_consensus.cc:493] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:57:16.519299 12443 raft_consensus.cc:3060] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:16.550110 12443 raft_consensus.cc:515] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.554107 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab1ce3df47c44c9bb38768bc4c44d7f" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56"
I20260926 01:57:16.554869 12026 raft_consensus.cc:3060] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:16.565152 12026 raft_consensus.cc:2468] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 1.
I20260926 01:57:16.568617 12171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab1ce3df47c44c9bb38768bc4c44d7f" candidate_uuid: "495ab022dafc493395ae0e21d9a77ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b877862e4a543229352921130f57068"
I20260926 01:57:16.569336 12171 raft_consensus.cc:3060] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:16.597213 12171 raft_consensus.cc:2468] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495ab022dafc493395ae0e21d9a77ee3 in term 1.
I20260926 01:57:16.602631 12443 leader_election.cc:290] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 election: Requested vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:16.604841 12252 leader_election.cc:304] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495ab022dafc493395ae0e21d9a77ee3, 5d58342c1ed14ef9a6815b9b5a219b56; no voters:
I20260926 01:57:16.606036 12443 raft_consensus.cc:2804] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:57:16.618409 12443 raft_consensus.cc:697] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Running, Role: LEADER
I20260926 01:57:16.621380 12524 raft_consensus.cc:493] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 01:57:16.635851 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:57:16.619418 12443 consensus_queue.cc:237] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.622363 12524 raft_consensus.cc:515] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:16.657395 12104 leader_election.cc:343] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: dde7568a319f44dbb455621070675a23
I20260926 01:57:16.658996 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dde7568a319f44dbb455621070675a23" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" is_pre_election: true
I20260926 01:57:16.659969 12026 raft_consensus.cc:2468] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b877862e4a543229352921130f57068 in term 0.
I20260926 01:57:16.661661 12107 leader_election.cc:304] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b877862e4a543229352921130f57068, 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:16.670316 12526 raft_consensus.cc:2804] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:57:16.670746 12526 raft_consensus.cc:493] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:57:16.671061 12526 raft_consensus.cc:3060] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:16.684901 12524 leader_election.cc:290] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:16.693557 12526 raft_consensus.cc:515] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.697372 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dde7568a319f44dbb455621070675a23" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56"
I20260926 01:57:16.698076 12026 raft_consensus.cc:3060] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:16.703709 12526 leader_election.cc:290] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 election: Requested vote from peers 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:16.705662 12026 raft_consensus.cc:2468] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b877862e4a543229352921130f57068 in term 1.
I20260926 01:57:16.707181 12107 leader_election.cc:304] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b877862e4a543229352921130f57068, 5d58342c1ed14ef9a6815b9b5a219b56; no voters:
I20260926 01:57:16.714670 12526 raft_consensus.cc:2804] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Leader election won for term 1
W20260926 01:57:16.719317 12104 leader_election.cc:343] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: dde7568a319f44dbb455621070675a23
I20260926 01:57:16.722234 12526 raft_consensus.cc:697] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 1 LEADER]: Becoming Leader. State: Replica: 1b877862e4a543229352921130f57068, State: Running, Role: LEADER
I20260926 01:57:16.723258 12526 consensus_queue.cc:237] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:16.728600 12482 consensus_queue.cc:1048] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:57:16.903831 12534 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:16.904657 12534 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:16.923535 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:16.930074 12443 consensus_queue.cc:1048] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:57:16.955482 12537 consensus_queue.cc:1048] T b554bcce374d4e5fa02957cd8aaebde8 P 495ab022dafc493395ae0e21d9a77ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:57:16.971082 12542 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9760316a8f74e829b4c7884b875b70e)
I20260926 01:57:16.971634 12542 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:17.003926 12542 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:17.020606 11728 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:17.024549 11730 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
W20260926 01:57:17.023818 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:17.025035 11730 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:17.025337 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:17.026125 12542 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260926 01:57:17.026605 12482 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:17.029736 12534 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:17.461350 12508 consensus_queue.cc:1048] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260926 01:57:17.610585 12104 consensus_peers.cc:597] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 -> Peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Couldn't send request to peer 495ab022dafc493395ae0e21d9a77ee3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dde7568a319f44dbb455621070675a23. This is attempt 1: this message will repeat every 5th retry.
I20260926 01:57:17.615690 12551 consensus_queue.cc:1048] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:57:17.803161 12557 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9760316a8f74e829b4c7884b875b70e)
I20260926 01:57:17.806981 12557 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
W20260926 01:57:17.823557 11730 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
I20260926 01:57:17.825641 12557 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:17.826720 12526 consensus_queue.cc:1048] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 01:57:17.842603 11728 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:17.843123 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6819a8c0ba75415c9cf089983ff5c191; no voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:17.844035 12557 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260926 01:57:17.647763 12512 debug-util.cc:398] Leaking SignalData structure 0x7b08000283c0 after lost signal to thread 12421
W20260926 01:57:18.097988 12512 kernel_stack_watchdog.cc:198] Thread 12421 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 395ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260926 01:57:16.250655 12514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:18.098819 12421 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.849s user 0.574s sys 0.996s
W20260926 01:57:18.099201 12421 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.849s user 0.574s sys 0.996s
W20260926 01:57:18.100780 12516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:18.102874 12515 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1848 milliseconds
I20260926 01:57:18.102952 12421 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:57:18.104306 12421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:57:18.107053 12421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:57:18.108403 12421 hybrid_clock.cc:648] HybridClock initialized: now 1790387838108372 us; error 46 us; skew 500 ppm
I20260926 01:57:18.109270 12421 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:57:18.116888 12421 webserver.cc:533] Webserver started at http://127.8.76.188:46041/ using document root <none> and password file <none>
I20260926 01:57:18.118213 12421 fs_manager.cc:362] Metadata directory not provided
I20260926 01:57:18.118507 12421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:57:18.127573 12421 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.002s
I20260926 01:57:18.132753 12568 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:18.134155 12421 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260926 01:57:18.134584 12421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
uuid: "596cdfef9f194805948e35ffe299ed65"
format_stamp: "Formatted at 2026-09-26 01:57:02 on dist-test-slave-5pk1"
I20260926 01:57:18.136677 12421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:57:18.185942 12421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:57:18.187918 12421 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:57:18.188409 12421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:57:18.276186 12421 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.188:36403
I20260926 01:57:18.276268 12619 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.188:36403 every 8 connection(s)
I20260926 01:57:18.279263 12421 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-2/data/info.pb
I20260926 01:57:18.282582 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 12421
I20260926 01:57:18.293038 12620 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:57:18.300247 12620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Bootstrap starting.
I20260926 01:57:18.318962 12620 log.cc:826] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Log is configured to *not* fsync() on all Append() calls
I20260926 01:57:18.335767 12620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:57:18.336678 12620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Bootstrap complete.
I20260926 01:57:18.357240 12362 heartbeater.cc:344] Connected to a master server at 127.8.76.188:36403
I20260926 01:57:18.357815 12362 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:18.361192 12620 raft_consensus.cc:359] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:18.363830 12620 raft_consensus.cc:740] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596cdfef9f194805948e35ffe299ed65, State: Initialized, Role: FOLLOWER
I20260926 01:57:18.364732 12620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:18.367354 12625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:18.368001 12625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:18.368899 12620 sys_catalog.cc:565] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:57:18.393043 12362 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:18.434350 12216 heartbeater.cc:344] Connected to a master server at 127.8.76.188:36403
I20260926 01:57:18.439251 12071 heartbeater.cc:344] Connected to a master server at 127.8.76.188:36403
I20260926 01:57:18.450906 12643 ts_tablet_manager.cc:933] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Initiating tablet copy from peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:18.456832 12643 tablet_copy_client.cc:323] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.76.130:34281
I20260926 01:57:18.459226 12191 tablet_copy_service.cc:140] P 1b877862e4a543229352921130f57068: Received BeginTabletCopySession request for tablet dde7568a319f44dbb455621070675a23 from peer 495ab022dafc493395ae0e21d9a77ee3 ({username='slave'} at 127.8.76.131:36917)
I20260926 01:57:18.459822 12191 tablet_copy_service.cc:161] P 1b877862e4a543229352921130f57068: Beginning new tablet copy session on tablet dde7568a319f44dbb455621070675a23 from peer 495ab022dafc493395ae0e21d9a77ee3 at {username='slave'} at 127.8.76.131:36917: session id = 495ab022dafc493395ae0e21d9a77ee3-dde7568a319f44dbb455621070675a23
I20260926 01:57:18.473541 12191 tablet_copy_source_session.cc:215] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 01:57:18.475299 12644 catalog_manager.cc:1308] Loaded cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:18.475663 12644 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: loading cluster ID for follower catalog manager: success
I20260926 01:57:18.480062 12643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde7568a319f44dbb455621070675a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:18.483808 12644 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: acquiring CA information for follower catalog manager: success
I20260926 01:57:18.489997 12644 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:57:18.503321 12643 tablet_copy_client.cc:806] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: tablet copy: Starting download of 0 data blocks...
I20260926 01:57:18.504148 12643 tablet_copy_client.cc:670] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: tablet copy: Starting download of 1 WAL segments...
I20260926 01:57:18.509083 12643 tablet_copy_client.cc:538] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 01:57:18.518882 12643 tablet_bootstrap.cc:492] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:18.494109 12421 master_runner.cc:187] Error getting master registration for 127.8.76.190:41989: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111),
I20260926 01:57:18.522032 12421 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:18.537963 12643 tablet_bootstrap.cc:492] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:57:18.538956 12643 tablet_bootstrap.cc:492] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap complete.
I20260926 01:57:18.539631 12643 ts_tablet_manager.cc:1403] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.001s
I20260926 01:57:18.542821 12643 raft_consensus.cc:359] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:18.543558 12643 raft_consensus.cc:740] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:18.544417 12643 consensus_queue.cc:260] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:18.545861 12362 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:18.546674 12643 ts_tablet_manager.cc:1434] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260926 01:57:18.549800 12191 tablet_copy_service.cc:342] P 1b877862e4a543229352921130f57068: Request end of tablet copy session 495ab022dafc493395ae0e21d9a77ee3-dde7568a319f44dbb455621070675a23 received from {username='slave'} at 127.8.76.131:36917
I20260926 01:57:18.550287 12191 tablet_copy_service.cc:434] P 1b877862e4a543229352921130f57068: ending tablet copy session 495ab022dafc493395ae0e21d9a77ee3-dde7568a319f44dbb455621070675a23 on tablet dde7568a319f44dbb455621070675a23 with peer 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:18.555418 12649 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9760316a8f74e829b4c7884b875b70e)
I20260926 01:57:18.555902 12584 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" instance_seqno: 1790387832687163) as {username='slave'} at 127.8.76.131:34551; Asking this server to re-register.
I20260926 01:57:18.555953 12649 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:18.558012 12362 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:18.560077 12649 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:18.566529 11730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260926 01:57:18.569460 12585 ts_manager.cc:194] Registered new tserver with Master: 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:18.582721 12595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
W20260926 01:57:18.583838 11730 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
I20260926 01:57:18.583894 12595 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6819a8c0ba75415c9cf089983ff5c191 in term 1.
I20260926 01:57:18.585434 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191; no voters: d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:18.586285 12649 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 01:57:18.586593 12649 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d9760316a8f74e829b4c7884b875b70e)
I20260926 01:57:18.586900 12649 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:57:18.593179 12649 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:18.594934 12649 leader_election.cc:290] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 election: Requested vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
I20260926 01:57:18.596217 12595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6819a8c0ba75415c9cf089983ff5c191" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65"
I20260926 01:57:18.596688 12595 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 1 FOLLOWER]: Advancing to term 3
W20260926 01:57:18.602747 11730 leader_election.cc:336] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
I20260926 01:57:18.606051 12595 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6819a8c0ba75415c9cf089983ff5c191 in term 3.
I20260926 01:57:18.609736 11728 leader_election.cc:304] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191; no voters: d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:18.610651 12649 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 01:57:18.615145 12649 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [term 3 LEADER]: Becoming Leader. State: Replica: 6819a8c0ba75415c9cf089983ff5c191, State: Running, Role: LEADER
I20260926 01:57:18.616238 12649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:18.621937 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6819a8c0ba75415c9cf089983ff5c191. Latest consensus state: current_term: 3 leader_uuid: "6819a8c0ba75415c9cf089983ff5c191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:18.622776 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: This master's current role is: LEADER
I20260926 01:57:18.635924 12656 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:57:18.663177 12656 catalog_manager.cc:692] Loaded metadata for table default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]
I20260926 01:57:18.666796 12656 catalog_manager.cc:692] Loaded metadata for table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]
I20260926 01:57:18.668902 12656 catalog_manager.cc:692] Loaded metadata for table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]
I20260926 01:57:18.689915 12583 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" instance_seqno: 1790387828113500) as {username='slave'} at 127.8.76.129:40341; Asking this server to re-register.
I20260926 01:57:18.690825 12656 tablet_loader.cc:96] loaded metadata for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec])
I20260926 01:57:18.692291 12071 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:18.694682 12656 tablet_loader.cc:96] loaded metadata for tablet b554bcce374d4e5fa02957cd8aaebde8 (table default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236])
I20260926 01:57:18.697172 12585 ts_manager.cc:194] Registered new tserver with Master: 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369)
I20260926 01:57:18.697027 12656 tablet_loader.cc:96] loaded metadata for tablet dde7568a319f44dbb455621070675a23 (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122])
I20260926 01:57:18.700387 12656 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:57:18.705540 12656 catalog_manager.cc:1308] Loaded cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:18.705862 12656 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:57:18.711517 12656 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:57:18.716357 12656 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: Loaded TSK: 0
I20260926 01:57:18.717906 12656 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:57:18.722761 12656 catalog_manager.cc:1576] Loading latest processed Hive Metastore notification log event ID...
I20260926 01:57:18.727408 12656 catalog_manager.cc:6068] Last processed Hive Metastore notification event ID: -1
I20260926 01:57:18.742007 12595 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Refusing update from remote peer 6819a8c0ba75415c9cf089983ff5c191: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20260926 01:57:18.744067 12654 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 4, Time since last communication: 0.000s
W20260926 01:57:18.746141 11730 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 -> Peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Couldn't send request to peer d9760316a8f74e829b4c7884b875b70e. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:57:18.770661 11744 master_service.cc:960] ReplaceTablet: received request to replace tablet 5ab9e06a641740c7bbb1afbe0f4943a0 from {username='slave'} at 127.0.0.1:40168
I20260926 01:57:18.773223 11744 master_service.cc:960] ReplaceTablet: received request to replace tablet dde7568a319f44dbb455621070675a23 from {username='slave'} at 127.0.0.1:40168
I20260926 01:57:18.784178 11743 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40168:
name: "default.Table_df54eb8e69094653a7e2b65f39cea88c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:18.787642 11742 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40168:
name: "default.Table_9aa9a3fdad284796b3f433da53236871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:18.790792 11741 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40168:
name: "default.Table_5c81b19f115f450fb219ec996c8c470b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:18.793844 11740 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40168:
table { table_name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168" } modify_external_catalogs: true
I20260926 01:57:18.794921 11740 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40168:
table { table_name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168" } modify_external_catalogs: true
I20260926 01:57:18.797253 11739 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40168:
table { table_name: "default.Table_5bdf5515f5af48e1bd0f78249a93cedc" } new_table_name: "default.Table_9696800ab3c24a7a911de7daea878289" modify_external_catalogs: true
I20260926 01:57:18.799875 11738 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40168:
table { table_name: "default.Table_9ecb0b51e0e947d4bc9d0fb60139994f" } new_table_name: "default.Table_75ff2d45355c4608803c3287943fdb45" modify_external_catalogs: true
I20260926 01:57:18.802479 11737 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40168:
name: "default.Table_09a88a4aa04c44fd9ff3aac5e421f67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260926 01:57:18.811019 11897 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1790387833, eventType=CREATE_TABLE, dbName=default, tableName=table_7440abc68fd740d49170b27f2cb1c731, message={"server":"","servicePrincipal":"","db":"default","table":"table_7440abc68fd740d49170b27f2cb1c731","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7440abc68fd740d49170b27f2cb1c731\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790387833},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/warehouse/table_7440abc68fd740d49170b27f2cb1c731\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7440abc68fd740d49170b27f2cb1c731\",\"kudu.table_id\":\"a49a2e5756be4c1384ad85aba0834a58\",\"transient_lastDdlTime\":\"1790387833\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"29d8d2959c5241009da641125beda276\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1790387833,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1790387833, eventType=CREATE_TABLE, dbName=default, tableName=table_9ecb0b51e0e947d4bc9d0fb60139994f, message={"server":"","servicePrincipal":"","db":"default","table":"table_9ecb0b51e0e947d4bc9d0fb60139994f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9ecb0b51e0e947d4bc9d0fb60139994f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790387833},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/warehouse/table_9ecb0b51e0e947d4bc9d0fb60139994f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9ecb0b51e0e947d4bc9d0fb60139994f\",\"kudu.table_id\":\"c3c32452d955463485f7f60723c019ec\",\"transient_lastDdlTime\":\"1790387833\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"29d8d2959c5241009da641125beda276\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1790387833,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f86bfa00920 google::LogMessage::Flush() at ??:0
@ 0x7f86bfa0068f google::LogMessage::~LogMessage() at ??:0
I20260926 01:57:18.878916 12663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6819a8c0ba75415c9cf089983ff5c191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:18.879801 12663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:18.880115 12625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6819a8c0ba75415c9cf089983ff5c191. Latest consensus state: current_term: 3 leader_uuid: "6819a8c0ba75415c9cf089983ff5c191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:18.880729 12625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:18.901779 12625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6819a8c0ba75415c9cf089983ff5c191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:18.902535 12625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:18.911485 12649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6819a8c0ba75415c9cf089983ff5c191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:18.912248 12649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191 [sys.catalog]: This master's current role is: LEADER
@ 0x7f86cab9335c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260926 01:57:18.931692 11744 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: Sending DeleteTablet for 3 replicas of tablet dde7568a319f44dbb455621070675a23
I20260926 01:57:18.950336 12583 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b877862e4a543229352921130f57068" instance_seqno: 1790387830573023) as {username='slave'} at 127.8.76.130:33739; Asking this server to re-register.
I20260926 01:57:18.953323 12216 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:18.955241 11744 catalog_manager.cc:6934] ReplaceTablet: tablet dde7568a319f44dbb455621070675a23 deleted and replaced by tablet 8bf20c5dcbe44eebb4164a0cecece363
I20260926 01:57:18.957439 12583 ts_manager.cc:194] Registered new tserver with Master: 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
@ 0x7f86cab92b1c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260926 01:57:18.974599 12661 mvcc.cc:204] Tried to move back new op lower bound from 7333428587470053376 to 7333428565535363072. Current Snapshot: MvccSnapshot[applied={T|T < 7333428587470053376 or (T in {7333428587470053376})}]
I20260926 01:57:18.986989 12295 tablet_service.cc:1558] Processing DeleteTablet for tablet dde7568a319f44dbb455621070675a23 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bf20c5dcbe44eebb4164a0cecece363) from {username='slave'} at 127.0.0.1:47912
I20260926 01:57:18.989014 12004 tablet_service.cc:1558] Processing DeleteTablet for tablet dde7568a319f44dbb455621070675a23 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bf20c5dcbe44eebb4164a0cecece363) from {username='slave'} at 127.0.0.1:59802
@ 0x7f86cab96b92 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260926 01:57:19.001574 12005 tablet_service.cc:1511] Processing CreateTablet for tablet 8bf20c5dcbe44eebb4164a0cecece363 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:19.003119 12005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bf20c5dcbe44eebb4164a0cecece363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:19.006390 12677 tablet_replica.cc:333] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
I20260926 01:57:19.007318 12677 raft_consensus.cc:2243] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:19.007854 12677 raft_consensus.cc:2272] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:19.011386 12677 ts_tablet_manager.cc:1916] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:19.017462 11744 master_service.cc:960] ReplaceTablet: received request to replace tablet 6ab1ce3df47c44c9bb38768bc4c44d7f from {username='slave'} at 127.0.0.1:40168
I20260926 01:57:19.021895 12295 tablet_service.cc:1511] Processing CreateTablet for tablet 8bf20c5dcbe44eebb4164a0cecece363 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:19.023134 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bf20c5dcbe44eebb4164a0cecece363. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f86cab96b4a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260926 01:57:19.028908 12677 ts_tablet_manager.cc:1929] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:57:19.029345 12677 log.cc:1199] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/dde7568a319f44dbb455621070675a23
I20260926 01:57:19.031203 12677 ts_tablet_manager.cc:1950] T dde7568a319f44dbb455621070675a23 P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
I20260926 01:57:19.038722 11728 catalog_manager.cc:5063] TS 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): tablet dde7568a319f44dbb455621070675a23 (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:19.039033 12150 tablet_service.cc:1558] Processing DeleteTablet for tablet dde7568a319f44dbb455621070675a23 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bf20c5dcbe44eebb4164a0cecece363) from {username='slave'} at 127.0.0.1:56734
I20260926 01:57:19.041391 12151 tablet_service.cc:1511] Processing CreateTablet for tablet 8bf20c5dcbe44eebb4164a0cecece363 (DEFAULT_TABLE table=default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:19.042990 12151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bf20c5dcbe44eebb4164a0cecece363. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f86cab96ada std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260926 01:57:19.059484 12678 tablet_replica.cc:333] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: stopping tablet replica
I20260926 01:57:19.060441 12678 raft_consensus.cc:2243] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:19.061038 12678 raft_consensus.cc:2272] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f86cab96aa2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260926 01:57:19.068747 12678 ts_tablet_manager.cc:1916] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:19.075721 11744 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 6819a8c0ba75415c9cf089983ff5c191: Sending DeleteTablet for 3 replicas of tablet 6ab1ce3df47c44c9bb38768bc4c44d7f
I20260926 01:57:19.077251 11744 catalog_manager.cc:6934] ReplaceTablet: tablet 6ab1ce3df47c44c9bb38768bc4c44d7f deleted and replaced by tablet 5782394e04914feca46251593cee424c
I20260926 01:57:19.079778 12004 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f with delete_type TABLET_DATA_DELETED (replaced by tablet 5782394e04914feca46251593cee424c) from {username='slave'} at 127.0.0.1:59802
@ 0x7f86cab95d9e std::__1::__function::__func<>::operator()() at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
I20260926 01:57:19.090804 12148 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f with delete_type TABLET_DATA_DELETED (replaced by tablet 5782394e04914feca46251593cee424c) from {username='slave'} at 127.0.0.1:56734
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
I20260926 01:57:19.093385 12296 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ab1ce3df47c44c9bb38768bc4c44d7f with delete_type TABLET_DATA_DELETED (replaced by tablet 5782394e04914feca46251593cee424c) from {username='slave'} at 127.0.0.1:47912
I20260926 01:57:19.094249 12677 tablet_replica.cc:333] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: stopping tablet replica
I20260926 01:57:19.095110 12677 raft_consensus.cc:2243] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 01:57:19.096307 12677 raft_consensus.cc:2272] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:19.097748 12683 tablet_bootstrap.cc:492] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:19.100286 12677 ts_tablet_manager.cc:1916] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:19.105392 12680 tablet_replica.cc:333] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:19.106251 12680 raft_consensus.cc:2243] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 1 LEADER]: Raft consensus shutting down.
@ 0x7f86c1bf681b kudu::Thread::SuperviseThread() at ??:0
@ 0x4569a0 __tsan_thread_start_func at ??:?
I20260926 01:57:19.111176 12680 raft_consensus.cc:2272] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:19.117949 12678 ts_tablet_manager.cc:1929] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:57:19.124275 12678 log.cc:1199] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/dde7568a319f44dbb455621070675a23
I20260926 01:57:19.125157 12678 ts_tablet_manager.cc:1950] T dde7568a319f44dbb455621070675a23 P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting consensus metadata
I20260926 01:57:19.126011 12683 tablet_bootstrap.cc:654] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
@ 0x7f86c33576db start_thread at ??:0
I20260926 01:57:19.128795 11731 catalog_manager.cc:5063] TS 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369): tablet dde7568a319f44dbb455621070675a23 (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]) successfully deleted
I20260926 01:57:19.130769 12677 ts_tablet_manager.cc:1929] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:57:19.131265 12677 log.cc:1199] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-2/wal/wals/6ab1ce3df47c44c9bb38768bc4c44d7f
I20260926 01:57:19.133546 12150 tablet_service.cc:1511] Processing CreateTablet for tablet 5782394e04914feca46251593cee424c (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:19.134904 12150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5782394e04914feca46251593cee424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:19.135990 12004 tablet_service.cc:1511] Processing CreateTablet for tablet 5782394e04914feca46251593cee424c (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:19.137169 12004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5782394e04914feca46251593cee424c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 01:57:19.137768 12295 tablet_service.cc:1511] Processing CreateTablet for tablet 5782394e04914feca46251593cee424c (DEFAULT_TABLE table=default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 01:57:19.138725 12680 ts_tablet_manager.cc:1916] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:19.138926 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5782394e04914feca46251593cee424c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f86be6a261f clone at ??:0
I20260926 01:57:19.151929 12677 ts_tablet_manager.cc:1950] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 495ab022dafc493395ae0e21d9a77ee3: Deleting consensus metadata
I20260926 01:57:19.155391 12678 tablet_replica.cc:333] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: stopping tablet replica
I20260926 01:57:19.156280 12678 raft_consensus.cc:2243] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:19.159013 12686 tablet_bootstrap.cc:492] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:19.160020 12688 tablet_bootstrap.cc:492] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:19.175029 12688 tablet_bootstrap.cc:654] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:19.177488 12680 ts_tablet_manager.cc:1929] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:57:19.177987 12680 log.cc:1199] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/dde7568a319f44dbb455621070675a23
I20260926 01:57:19.170315 12678 raft_consensus.cc:2272] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:19.179065 12680 ts_tablet_manager.cc:1950] T dde7568a319f44dbb455621070675a23 P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:19.182314 12680 tablet_replica.cc:333] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: stopping tablet replica
I20260926 01:57:19.183071 12680 raft_consensus.cc:2243] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 01:57:19.184564 12680 raft_consensus.cc:2272] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 01:57:19.186851 12680 ts_tablet_manager.cc:1916] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:19.199681 12688 tablet_bootstrap.cc:492] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:19.200237 12688 ts_tablet_manager.cc:1403] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.000s
I20260926 01:57:19.202701 12683 tablet_bootstrap.cc:492] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:19.203189 12683 ts_tablet_manager.cc:1403] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.004s
I20260926 01:57:19.204159 12680 ts_tablet_manager.cc:1929] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:57:19.204619 12680 log.cc:1199] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-1/wal/wals/6ab1ce3df47c44c9bb38768bc4c44d7f
I20260926 01:57:19.205386 12680 ts_tablet_manager.cc:1950] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 1b877862e4a543229352921130f57068: Deleting consensus metadata
I20260926 01:57:19.205045 12688 raft_consensus.cc:359] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.205862 12688 raft_consensus.cc:385] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:19.206153 12688 raft_consensus.cc:740] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:19.205945 12683 raft_consensus.cc:359] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.206651 12683 raft_consensus.cc:385] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:19.206918 12683 raft_consensus.cc:740] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:19.206969 12688 consensus_queue.cc:260] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.209599 12688 ts_tablet_manager.cc:1434] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 01:57:19.210548 12688 tablet_bootstrap.cc:492] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068: Bootstrap starting.
I20260926 01:57:19.210738 12217 heartbeater.cc:499] Master 127.8.76.189:34661 was elected leader, sending a full tablet report...
I20260926 01:57:19.217303 12688 tablet_bootstrap.cc:654] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:19.220021 12683 consensus_queue.cc:260] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.226250 12363 heartbeater.cc:499] Master 127.8.76.189:34661 was elected leader, sending a full tablet report...
I20260926 01:57:19.229736 12683 ts_tablet_manager.cc:1434] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.026s user 0.002s sys 0.004s
I20260926 01:57:19.237174 12678 ts_tablet_manager.cc:1916] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 01:57:19.230729 12683 tablet_bootstrap.cc:492] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3: Bootstrap starting.
I20260926 01:57:19.250257 12683 tablet_bootstrap.cc:654] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:19.252152 12686 tablet_bootstrap.cc:654] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:19.272554 12686 tablet_bootstrap.cc:492] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:19.273036 12686 ts_tablet_manager.cc:1403] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.008s
I20260926 01:57:19.277961 12683 tablet_bootstrap.cc:492] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3: No bootstrap required, opened a new log
I20260926 01:57:19.278493 12683 ts_tablet_manager.cc:1403] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.008s
I20260926 01:57:19.289880 12688 tablet_bootstrap.cc:492] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068: No bootstrap required, opened a new log
I20260926 01:57:19.290397 12688 ts_tablet_manager.cc:1403] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068: Time spent bootstrapping tablet: real 0.080s user 0.001s sys 0.016s
I20260926 01:57:19.291810 12683 raft_consensus.cc:359] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.292657 12683 raft_consensus.cc:385] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:19.292959 12683 raft_consensus.cc:740] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495ab022dafc493395ae0e21d9a77ee3, State: Initialized, Role: FOLLOWER
I20260926 01:57:19.293460 12688 raft_consensus.cc:359] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.293694 12683 consensus_queue.cc:260] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.294154 12688 raft_consensus.cc:385] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:19.295926 12683 ts_tablet_manager.cc:1434] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260926 01:57:19.298138 12678 ts_tablet_manager.cc:1929] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260926 01:57:19.298578 12678 log.cc:1199] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting WAL directory at /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/ts-0/wal/wals/6ab1ce3df47c44c9bb38768bc4c44d7f
I20260926 01:57:19.299355 12678 ts_tablet_manager.cc:1950] T 6ab1ce3df47c44c9bb38768bc4c44d7f P 5d58342c1ed14ef9a6815b9b5a219b56: Deleting consensus metadata
I20260926 01:57:19.303042 12686 raft_consensus.cc:359] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.303758 12686 raft_consensus.cc:385] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:19.304029 12686 raft_consensus.cc:740] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:19.294425 12688 raft_consensus.cc:740] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b877862e4a543229352921130f57068, State: Initialized, Role: FOLLOWER
I20260926 01:57:19.305512 12688 consensus_queue.cc:260] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.307803 12688 ts_tablet_manager.cc:1434] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068: Time spent starting tablet: real 0.017s user 0.000s sys 0.006s
I20260926 01:57:19.308682 12686 consensus_queue.cc:260] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.335526 12686 ts_tablet_manager.cc:1434] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.062s user 0.009s sys 0.019s
I20260926 01:57:19.339118 12686 tablet_bootstrap.cc:492] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56: Bootstrap starting.
I20260926 01:57:19.349601 12686 tablet_bootstrap.cc:654] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56: Neither blocks nor log segments found. Creating new log.
I20260926 01:57:19.356163 12686 tablet_bootstrap.cc:492] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56: No bootstrap required, opened a new log
I20260926 01:57:19.356657 12686 ts_tablet_manager.cc:1403] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.010s
I20260926 01:57:19.359592 12686 raft_consensus.cc:359] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.360316 12686 raft_consensus.cc:385] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 01:57:19.360602 12686 raft_consensus.cc:740] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Initialized, Role: FOLLOWER
I20260926 01:57:19.362674 12686 consensus_queue.cc:260] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.365036 12686 ts_tablet_manager.cc:1434] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260926 01:57:19.386790 12526 raft_consensus.cc:493] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:19.387380 12526 raft_consensus.cc:515] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.391523 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5782394e04914feca46251593cee424c" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" is_pre_election: true
I20260926 01:57:19.392252 12026 raft_consensus.cc:2468] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b877862e4a543229352921130f57068 in term 0.
I20260926 01:57:19.394374 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5782394e04914feca46251593cee424c" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3" is_pre_election: true
I20260926 01:57:19.395394 12316 raft_consensus.cc:2468] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b877862e4a543229352921130f57068 in term 0.
I20260926 01:57:19.397148 12104 leader_election.cc:304] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3; no voters:
I20260926 01:57:19.401276 12526 leader_election.cc:290] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707), 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369)
I20260926 01:57:19.402032 12526 raft_consensus.cc:2804] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:57:19.402503 12526 raft_consensus.cc:493] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:57:19.402928 12526 raft_consensus.cc:3060] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:19.411904 12526 raft_consensus.cc:515] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.416002 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5782394e04914feca46251593cee424c" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3"
I20260926 01:57:19.416689 12316 raft_consensus.cc:3060] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:19.426936 12316 raft_consensus.cc:2468] T 5782394e04914feca46251593cee424c P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b877862e4a543229352921130f57068 in term 1.
I20260926 01:57:19.429672 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5782394e04914feca46251593cee424c" candidate_uuid: "1b877862e4a543229352921130f57068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d58342c1ed14ef9a6815b9b5a219b56"
I20260926 01:57:19.430305 12026 raft_consensus.cc:3060] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:19.440758 12026 raft_consensus.cc:2468] T 5782394e04914feca46251593cee424c P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b877862e4a543229352921130f57068 in term 1.
I20260926 01:57:19.442346 12526 leader_election.cc:290] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 election: Requested vote from peers 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707), 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369)
I20260926 01:57:19.442663 12104 leader_election.cc:304] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3; no voters:
I20260926 01:57:19.443688 12526 raft_consensus.cc:2804] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:57:19.444242 12526 raft_consensus.cc:697] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [term 1 LEADER]: Becoming Leader. State: Replica: 1b877862e4a543229352921130f57068, State: Running, Role: LEADER
I20260926 01:57:19.445165 12526 consensus_queue.cc:237] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
W20260926 01:57:19.518970 12252 connection.cc:573] server connection from 127.0.0.1:47912 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:19.519164 12105 connection.cc:573] server connection from 127.0.0.1:56734 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:19.519811 12105 connection.cc:573] client connection to 127.8.76.189:34661 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:19.519996 12373 connection.cc:573] client connection to 127.8.76.189:34661 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:19.521934 12373 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.76.189:34661)
W20260926 01:57:19.523010 11962 connection.cc:573] server connection from 127.0.0.1:59802 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:19.523751 11960 connection.cc:573] client connection to 127.8.76.189:34661 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:19.519011 12250 connection.cc:573] client connection to 127.8.76.189:34661 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:19.526330 12552 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:19.522019 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.8.76.189:34661 (local address 127.8.76.130:36739)
0926 01:57:19.523195 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0926 01:57:19.523245 (+ 50us) client_negotiation.cc:175] Beginning negotiation
0926 01:57:19.523563 (+ 318us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0926 01:57:19.524956 (+ 1393us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":482}
W20260926 01:57:19.530620 12373 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.189:34661)
W20260926 01:57:19.531490 12373 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.76.189:34661)
W20260926 01:57:19.533000 12373 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.76.189:34661)
I20260926 01:57:19.636139 12705 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6819a8c0ba75415c9cf089983ff5c191)
I20260926 01:57:19.636737 12705 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:19.640064 12705 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
W20260926 01:57:19.651194 12571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
W20260926 01:57:19.661702 12571 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
W20260926 01:57:19.662400 12570 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:19.662787 12570 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65; no voters: 6819a8c0ba75415c9cf089983ff5c191, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:19.663637 12705 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260926 01:57:19.712321 12698 raft_consensus.cc:493] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:19.712890 12698 raft_consensus.cc:515] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.715130 12698 leader_election.cc:290] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:19.716904 12170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bf20c5dcbe44eebb4164a0cecece363" candidate_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b877862e4a543229352921130f57068" is_pre_election: true
I20260926 01:57:19.717180 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bf20c5dcbe44eebb4164a0cecece363" candidate_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3" is_pre_election: true
I20260926 01:57:19.717489 12170 raft_consensus.cc:2468] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d58342c1ed14ef9a6815b9b5a219b56 in term 0.
I20260926 01:57:19.717873 12316 raft_consensus.cc:2468] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d58342c1ed14ef9a6815b9b5a219b56 in term 0.
I20260926 01:57:19.718634 11959 leader_election.cc:304] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b877862e4a543229352921130f57068, 5d58342c1ed14ef9a6815b9b5a219b56; no voters:
I20260926 01:57:19.719350 12698 raft_consensus.cc:2804] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 01:57:19.719696 12698 raft_consensus.cc:493] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 01:57:19.720005 12698 raft_consensus.cc:3060] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:19.727372 12698 raft_consensus.cc:515] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:19.729444 12698 leader_election.cc:290] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 election: Requested vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:19.730511 12170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bf20c5dcbe44eebb4164a0cecece363" candidate_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b877862e4a543229352921130f57068"
I20260926 01:57:19.730886 12316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bf20c5dcbe44eebb4164a0cecece363" candidate_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495ab022dafc493395ae0e21d9a77ee3"
I20260926 01:57:19.731148 12170 raft_consensus.cc:3060] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:19.731456 12316 raft_consensus.cc:3060] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 01:57:19.738955 12170 raft_consensus.cc:2468] T 8bf20c5dcbe44eebb4164a0cecece363 P 1b877862e4a543229352921130f57068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d58342c1ed14ef9a6815b9b5a219b56 in term 1.
I20260926 01:57:19.740437 12316 raft_consensus.cc:2468] T 8bf20c5dcbe44eebb4164a0cecece363 P 495ab022dafc493395ae0e21d9a77ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d58342c1ed14ef9a6815b9b5a219b56 in term 1.
I20260926 01:57:19.740406 11959 leader_election.cc:304] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b877862e4a543229352921130f57068, 5d58342c1ed14ef9a6815b9b5a219b56; no voters:
I20260926 01:57:19.742025 12698 raft_consensus.cc:2804] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 01:57:19.747063 12698 raft_consensus.cc:697] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [term 1 LEADER]: Becoming Leader. State: Replica: 5d58342c1ed14ef9a6815b9b5a219b56, State: Running, Role: LEADER
I20260926 01:57:19.748500 12698 consensus_queue.cc:237] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } } peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } }
I20260926 01:57:20.214246 12715 consensus_queue.cc:1048] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:57:20.221500 12716 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6819a8c0ba75415c9cf089983ff5c191)
I20260926 01:57:20.222209 12716 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:20.226218 12711 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:20.226771 12711 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:20.228915 12711 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:20.231705 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:57:20.232369 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:20.232735 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:20.233984 12711 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:20.250072 12716 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
W20260926 01:57:20.258488 12571 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
W20260926 01:57:20.262812 12570 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:20.263315 12570 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65; no voters: 6819a8c0ba75415c9cf089983ff5c191, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:20.264071 12716 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260926 01:57:20.280406 12715 consensus_queue.cc:1048] T 5782394e04914feca46251593cee424c P 1b877862e4a543229352921130f57068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 01:57:20.691394 12711 consensus_queue.cc:1048] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:57:20.713253 12729 consensus_queue.cc:1048] T 8bf20c5dcbe44eebb4164a0cecece363 P 5d58342c1ed14ef9a6815b9b5a219b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 01:57:20.724872 12730 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:20.726100 12730 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:20.734333 12730 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:20.737100 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:57:20.739156 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:20.739583 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:20.743260 12730 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:21.236507 12735 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6819a8c0ba75415c9cf089983ff5c191)
I20260926 01:57:21.236958 12735 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:21.238803 12735 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
W20260926 01:57:21.245220 12571 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
W20260926 01:57:21.246282 12570 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:21.246742 12570 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65; no voters: 6819a8c0ba75415c9cf089983ff5c191, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:21.247481 12735 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260926 01:57:21.428534 8498 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
/tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.190:41989
--webserver_interface=127.8.76.190
--webserver_port=36691
--builtin_ntp_servers=127.8.76.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--hive_metastore_uris=thrift://127.0.0.1:43863
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260926 01:57:21.805454 12741 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 01:57:21.806185 12741 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 01:57:21.806756 12741 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260926 01:57:21.845894 12741 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260926 01:57:21.846185 12741 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260926 01:57:21.846508 12741 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 01:57:21.846793 12741 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260926 01:57:21.847046 12741 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 01:57:21.847294 12741 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260926 01:57:21.890575 12741 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.76.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43863
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.76.190:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.76.190
--webserver_port=36691
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type FASTDEBUG
built by None at 26 Sep 2026 01:43:22 UTC on e7f111948823
build id 14002
TSAN enabled
I20260926 01:57:21.892438 12741 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 01:57:21.894747 12741 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260926 01:57:21.909000 12753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 01:57:22.280263 12764 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6819a8c0ba75415c9cf089983ff5c191)
I20260926 01:57:22.282039 12764 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:22.286665 12764 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
W20260926 01:57:22.355336 12570 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:22.365248 12571 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989): Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111)
I20260926 01:57:22.368712 12571 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65; no voters: 6819a8c0ba75415c9cf089983ff5c191, d9760316a8f74e829b4c7884b875b70e
I20260926 01:57:22.380229 12764 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260926 01:57:23.307626 12774 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:23.308653 12774 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:23.313791 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:57:23.318625 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:23.319161 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:23.321357 12774 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:23.326615 12774 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:57:23.312457 12752 debug-util.cc:398] Leaking SignalData structure 0x7b08000283c0 after lost signal to thread 12741
W20260926 01:57:23.337869 12741 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.429s user 0.474s sys 0.801s
W20260926 01:57:23.338315 12741 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.429s user 0.474s sys 0.801s
W20260926 01:57:21.909525 12754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:23.342048 12756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 01:57:23.347015 12755 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1435 milliseconds
I20260926 01:57:23.347060 12741 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260926 01:57:23.348516 12741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 01:57:23.351606 12741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 01:57:23.352970 12741 hybrid_clock.cc:648] HybridClock initialized: now 1790387843352934 us; error 49 us; skew 500 ppm
I20260926 01:57:23.353834 12741 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 01:57:23.360709 12741 webserver.cc:533] Webserver started at http://127.8.76.190:36691/ using document root <none> and password file <none>
I20260926 01:57:23.361815 12741 fs_manager.cc:362] Metadata directory not provided
I20260926 01:57:23.362085 12741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 01:57:23.372969 12741 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.002s
I20260926 01:57:23.379268 12781 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 01:57:23.380738 12741 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 01:57:23.381111 12741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data,/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
uuid: "d9760316a8f74e829b4c7884b875b70e"
format_stamp: "Formatted at 2026-09-26 01:56:58 on dist-test-slave-5pk1"
I20260926 01:57:23.383262 12741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 01:57:23.470151 12741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260926 01:57:23.472082 12741 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 01:57:23.472528 12741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 01:57:23.525882 12421 master_runner.cc:187] Error getting master registration for 127.8.76.190:41989: Network error: Client connection negotiation failed: client connection to 127.8.76.190:41989: connect: Connection refused (error 111),
I20260926 01:57:23.526361 12421 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:23.560833 12741 rpc_server.cc:307] RPC server started. Bound to: 127.8.76.190:41989
I20260926 01:57:23.561007 12836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.76.190:41989 every 8 connection(s)
I20260926 01:57:23.564071 12741 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/master-0/data/info.pb
I20260926 01:57:23.568501 8498 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskiRTH_v/build/tsan/bin/kudu as pid 12741
I20260926 01:57:23.581260 12837 sys_catalog.cc:263] Verifying existing consensus state
I20260926 01:57:23.589596 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Bootstrap starting.
I20260926 01:57:23.616886 12837 log.cc:826] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Log is configured to *not* fsync() on all Append() calls
I20260926 01:57:23.689258 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=16 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 01:57:23.690119 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: Bootstrap complete.
I20260926 01:57:23.708829 12837 raft_consensus.cc:359] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:23.711258 12837 raft_consensus.cc:740] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9760316a8f74e829b4c7884b875b70e, State: Initialized, Role: FOLLOWER
I20260926 01:57:23.712090 12837 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:23.715039 12841 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:23.715731 12841 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:23.715871 12837 sys_catalog.cc:565] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 01:57:23.772660 12364 heartbeater.cc:344] Connected to a master server at 127.8.76.190:41989
I20260926 01:57:23.773442 12856 catalog_manager.cc:1308] Loaded cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:23.773715 12856 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: loading cluster ID for follower catalog manager: success
I20260926 01:57:23.784390 12801 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" instance_seqno: 1790387832687163) as {username='slave'} at 127.8.76.131:41877; Asking this server to re-register.
I20260926 01:57:23.786618 12364 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:23.788381 12856 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: acquiring CA information for follower catalog manager: success
I20260926 01:57:23.792654 12801 ts_manager.cc:194] Registered new tserver with Master: 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:23.809737 12856 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 01:57:23.821162 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:23.849726 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:23.947633 12075 heartbeater.cc:344] Connected to a master server at 127.8.76.190:41989
I20260926 01:57:23.952405 12801 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" instance_seqno: 1790387828113500) as {username='slave'} at 127.8.76.129:51141; Asking this server to re-register.
I20260926 01:57:23.953801 12075 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:23.957099 12801 ts_manager.cc:194] Registered new tserver with Master: 5d58342c1ed14ef9a6815b9b5a219b56 (127.8.76.129:41369)
I20260926 01:57:24.064580 12218 heartbeater.cc:344] Connected to a master server at 127.8.76.190:41989
I20260926 01:57:24.068634 12801 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b877862e4a543229352921130f57068" instance_seqno: 1790387830573023) as {username='slave'} at 127.8.76.130:52605; Asking this server to re-register.
I20260926 01:57:24.070006 12218 heartbeater.cc:461] Registering TS with master...
I20260926 01:57:24.073407 12801 ts_manager.cc:194] Registered new tserver with Master: 1b877862e4a543229352921130f57068 (127.8.76.130:34281)
I20260926 01:57:24.286711 12863 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:24.287312 12863 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:24.290380 12863 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:24.294179 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:24.322381 12595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9760316a8f74e829b4c7884b875b70e" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: false dest_uuid: "596cdfef9f194805948e35ffe299ed65" is_pre_election: true
W20260926 01:57:24.322365 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:24.323177 12595 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9760316a8f74e829b4c7884b875b70e in current term 3: Already voted for candidate 6819a8c0ba75415c9cf089983ff5c191 in this term.
I20260926 01:57:24.324862 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:24.325883 12863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 2 FOLLOWER]: Advancing to term 3
I20260926 01:57:24.332733 12863 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260926 01:57:24.408819 12866 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6819a8c0ba75415c9cf089983ff5c191)
I20260926 01:57:24.409262 12866 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:24.411016 12866 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
W20260926 01:57:24.433552 12570 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:24.437592 12811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "596cdfef9f194805948e35ffe299ed65" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e" is_pre_election: true
I20260926 01:57:24.438421 12811 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596cdfef9f194805948e35ffe299ed65 in term 3.
I20260926 01:57:24.439857 12571 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e; no voters: 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:24.440721 12866 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260926 01:57:24.441012 12866 raft_consensus.cc:493] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6819a8c0ba75415c9cf089983ff5c191)
I20260926 01:57:24.441280 12866 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 3 FOLLOWER]: Advancing to term 4
I20260926 01:57:24.448750 12866 raft_consensus.cc:515] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:24.451330 12866 leader_election.cc:290] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 election: Requested vote from peers d9760316a8f74e829b4c7884b875b70e (127.8.76.190:41989), 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661)
I20260926 01:57:24.452219 12811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "596cdfef9f194805948e35ffe299ed65" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "d9760316a8f74e829b4c7884b875b70e"
I20260926 01:57:24.452832 12811 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 3 FOLLOWER]: Advancing to term 4
W20260926 01:57:24.456717 12570 leader_election.cc:336] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:24.459072 12811 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596cdfef9f194805948e35ffe299ed65 in term 4.
I20260926 01:57:24.461357 12571 leader_election.cc:304] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 596cdfef9f194805948e35ffe299ed65, d9760316a8f74e829b4c7884b875b70e; no voters: 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:24.462561 12866 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 4 FOLLOWER]: Leader election won for term 4
I20260926 01:57:24.465399 12866 raft_consensus.cc:697] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [term 4 LEADER]: Becoming Leader. State: Replica: 596cdfef9f194805948e35ffe299ed65, State: Running, Role: LEADER
I20260926 01:57:24.466511 12866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:24.471990 12872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596cdfef9f194805948e35ffe299ed65. Latest consensus state: current_term: 4 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:24.472891 12872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: LEADER
I20260926 01:57:24.475400 12874 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260926 01:57:24.481613 12874 catalog_manager.cc:692] Loaded metadata for table default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236]
I20260926 01:57:24.482971 12874 catalog_manager.cc:692] Loaded metadata for table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122]
I20260926 01:57:24.484550 12874 catalog_manager.cc:692] Loaded metadata for table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec]
I20260926 01:57:24.490953 12874 tablet_loader.cc:96] loaded metadata for tablet 5782394e04914feca46251593cee424c (table default.table_9ecb0b51e0e947d4bc9d0fb60139994f [id=c3c32452d955463485f7f60723c019ec])
I20260926 01:57:24.492477 12874 tablet_loader.cc:96] loaded metadata for tablet 8bf20c5dcbe44eebb4164a0cecece363 (table default.table_5bdf5515f5af48e1bd0f78249a93cedc [id=85a53b32a7774185b1152340e907f122])
I20260926 01:57:24.493413 12874 tablet_loader.cc:96] loaded metadata for tablet b554bcce374d4e5fa02957cd8aaebde8 (table default.table_47c5a1843d9a4eb2ab8eba451134b168 [id=8077aa6cba9a4d108d72edd3d90d4236])
I20260926 01:57:24.494313 12874 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260926 01:57:24.497929 12874 catalog_manager.cc:1308] Loaded cluster ID: 29d8d2959c5241009da641125beda276
I20260926 01:57:24.498149 12874 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260926 01:57:24.502995 12874 catalog_manager.cc:1553] Loading token signing keys...
I20260926 01:57:24.506851 12874 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Loaded TSK: 0
I20260926 01:57:24.508045 12874 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260926 01:57:24.511444 12874 catalog_manager.cc:1576] Loading latest processed Hive Metastore notification log event ID...
I20260926 01:57:24.515786 12874 catalog_manager.cc:6068] Last processed Hive Metastore notification event ID: -1
W20260926 01:57:24.521375 12583 catalog_manager.cc:5671] Ignoring report from unknown tablet 5ab9e06a641740c7bbb1afbe0f4943a0
I20260926 01:57:24.537051 12811 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Refusing update from remote peer 596cdfef9f194805948e35ffe299ed65: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20260926 01:57:24.538901 12872 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 25, Time since last communication: 0.000s
W20260926 01:57:24.539820 12570 consensus_peers.cc:597] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 -> Peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Couldn't send request to peer 6819a8c0ba75415c9cf089983ff5c191. Status: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 01:57:24.590783 12878 mvcc.cc:204] Tried to move back new op lower bound from 7333428588634906624 to 7333428586982928384. Current Snapshot: MvccSnapshot[applied={T|T < 7333428588634906624 or (T in {7333428588634906624})}]
I20260926 01:57:24.608595 12866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:24.609305 12866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65 [sys.catalog]: This master's current role is: LEADER
I20260926 01:57:24.603645 12880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:24.614039 12880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:24.617775 12583 master_service.cc:960] ReplaceTablet: received request to replace tablet 5ab9e06a641740c7bbb1afbe0f4943a0 from {username='slave'} at 127.0.0.1:46140
I20260926 01:57:24.616295 12863 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596cdfef9f194805948e35ffe299ed65. Latest consensus state: current_term: 4 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:24.618413 12863 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:24.620076 12583 master_service.cc:960] ReplaceTablet: received request to replace tablet 6ac6c587687c4d868586da53310dfa5b from {username='slave'} at 127.0.0.1:46140
I20260926 01:57:24.623083 12583 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46140:
name: "default.Table_09a88a4aa04c44fd9ff3aac5e421f67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:24.623736 12880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "596cdfef9f194805948e35ffe299ed65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } } }
I20260926 01:57:24.624444 12880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 01:57:24.627494 12584 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46140:
table { table_name: "default.Table_5bdf5515f5af48e1bd0f78249a93cedc" } new_table_name: "default.Table_9696800ab3c24a7a911de7daea878289" modify_external_catalogs: true
I20260926 01:57:24.629899 12582 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46140:
table { table_name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168" } modify_external_catalogs: true
I20260926 01:57:24.630683 12582 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46140:
table { table_name: "default.Table_47c5a1843d9a4eb2ab8eba451134b168" } modify_external_catalogs: true
I20260926 01:57:24.631126 12585 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46140:
table { table_name: "default.Table_9ecb0b51e0e947d4bc9d0fb60139994f" } new_table_name: "default.Table_75ff2d45355c4608803c3287943fdb45" modify_external_catalogs: true
I20260926 01:57:24.632843 12581 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46140:
name: "default.Table_5c81b19f115f450fb219ec996c8c470b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:24.635427 12580 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46140:
name: "default.Table_9aa9a3fdad284796b3f433da53236871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 01:57:24.641233 12579 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46140:
name: "default.Table_df54eb8e69094653a7e2b65f39cea88c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260926 01:57:24.649027 12639 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1790387833, eventType=CREATE_TABLE, dbName=default, tableName=table_7440abc68fd740d49170b27f2cb1c731, message={"server":"","servicePrincipal":"","db":"default","table":"table_7440abc68fd740d49170b27f2cb1c731","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_7440abc68fd740d49170b27f2cb1c731\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790387833},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/warehouse/table_7440abc68fd740d49170b27f2cb1c731\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_7440abc68fd740d49170b27f2cb1c731\",\"kudu.table_id\":\"a49a2e5756be4c1384ad85aba0834a58\",\"transient_lastDdlTime\":\"1790387833\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"29d8d2959c5241009da641125beda276\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1790387833,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1790387833, eventType=CREATE_TABLE, dbName=default, tableName=table_9ecb0b51e0e947d4bc9d0fb60139994f, message={"server":"","servicePrincipal":"","db":"default","table":"table_9ecb0b51e0e947d4bc9d0fb60139994f","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9ecb0b51e0e947d4bc9d0fb60139994f\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790387833},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskiRTH_v/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790387723386151-8498-0/minicluster-data/warehouse/table_9ecb0b51e0e947d4bc9d0fb60139994f\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9ecb0b51e0e947d4bc9d0fb60139994f\",\"kudu.table_id\":\"c3c32452d955463485f7f60723c019ec\",\"transient_lastDdlTime\":\"1790387833\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"29d8d2959c5241009da641125beda276\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1790387833,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fb68b59c920 google::LogMessage::Flush() at ??:0
@ 0x7fb68b59c68f google::LogMessage::~LogMessage() at ??:0
@ 0x7fb69672f35c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fb69672eb1c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb696732b92 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fb696732b4a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fb696732ada std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fb696732aa2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fb696731d9e std::__1::__function::__func<>::operator()() at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
@ 0x7fb68d79281b kudu::Thread::SuperviseThread() at ??:0
@ 0x4569a0 __tsan_thread_start_func at ??:?
I20260926 01:57:24.859292 12577 master_service.cc:960] ReplaceTablet: received request to replace tablet 6ac6c587687c4d868586da53310dfa5b from {username='slave'} at 127.0.0.1:46140
I20260926 01:57:24.859308 12578 master_service.cc:960] ReplaceTablet: received request to replace tablet 5782394e04914feca46251593cee424c from {username='slave'} at 127.0.0.1:46140
@ 0x7fb68eef36db start_thread at ??:0
I20260926 01:57:24.880708 12578 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 596cdfef9f194805948e35ffe299ed65: Sending DeleteTablet for 3 replicas of tablet 5782394e04914feca46251593cee424c
I20260926 01:57:24.890805 12578 catalog_manager.cc:6934] ReplaceTablet: tablet 5782394e04914feca46251593cee424c deleted and replaced by tablet 1eeaaf943beb400aad19aaa88113244d
@ 0x7fb68a23e61f clone at ??:0
I20260926 01:57:25.023718 12071 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:25.039655 12216 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
W20260926 01:57:25.162770 12785 connection.cc:573] server connection from 127.0.0.1:32940 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:25.162662 12372 connection.cc:573] client connection to 127.8.76.188:36403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:25.164132 12902 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:24.892324 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.8.76.129:41369)
0926 01:57:24.915848 (+ 23524us) server_negotiation.cc:207] Beginning negotiation
0926 01:57:24.915878 (+ 30us) server_negotiation.cc:400] Waiting for connection header
0926 01:57:25.162362 (+246484us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35368: BlockingRecv error: recv got EOF from 127.0.0.1:35368 (error 108)
Metrics: {"server-negotiator.queue_time_us":22973,"thread_start_us":22654,"threads_started":1}
W20260926 01:57:25.165123 12372 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.76.188:36403)
W20260926 01:57:25.166342 12372 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.76.188:36403)
W20260926 01:57:25.168002 12372 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.76.188:36403)
W20260926 01:57:25.168640 12372 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.76.188:36403)
W20260926 01:57:25.170099 11959 connection.cc:573] client connection to 127.8.76.188:36403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:25.172808 12899 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:24.887344 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.8.76.130:34281)
0926 01:57:24.904417 (+ 17073us) server_negotiation.cc:207] Beginning negotiation
0926 01:57:24.904458 (+ 41us) server_negotiation.cc:400] Waiting for connection header
0926 01:57:24.904530 (+ 72us) server_negotiation.cc:408] Connection header received
0926 01:57:24.904727 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0926 01:57:24.904757 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0926 01:57:24.905052 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0926 01:57:24.905260 (+ 208us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0926 01:57:25.171570 (+266310us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53900: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":16366,"thread_start_us":16429,"threads_started":1}
W20260926 01:57:25.175170 12104 connection.cc:573] client connection to 127.8.76.188:36403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 01:57:25.175307 12901 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:24.888407 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.8.76.131:46707)
0926 01:57:24.906642 (+ 18235us) server_negotiation.cc:207] Beginning negotiation
0926 01:57:24.906673 (+ 31us) server_negotiation.cc:400] Waiting for connection header
0926 01:57:24.906727 (+ 54us) server_negotiation.cc:408] Connection header received
0926 01:57:24.906891 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0926 01:57:24.906917 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0926 01:57:24.907210 (+ 293us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0926 01:57:24.907404 (+ 194us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0926 01:57:25.166884 (+259480us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41572: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":17586,"thread_start_us":21136,"threads_started":1}
W20260926 01:57:25.181537 12906 negotiation.cc:336] Failed RPC negotiation. Trace:
0926 01:57:25.177053 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.8.76.188:36403 (local address 127.0.0.1:46984)
0926 01:57:25.180379 (+ 3326us) negotiation.cc:107] Waiting for socket to connect
0926 01:57:25.180580 (+ 201us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":2701,"threads_started":1}
I20260926 01:57:25.287691 12362 heartbeater.cc:499] Master 127.8.76.188:36403 was elected leader, sending a full tablet report...
I20260926 01:57:25.404798 12912 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:25.405236 12912 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:25.415796 12912 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:25.420615 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:25.421250 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:25.421686 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:25.422441 12912 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:57:25.939844 12921 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:25.940388 12921 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:25.942436 12921 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:25.949013 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:25.954245 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:25.954959 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:25.955981 12921 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:57:26.207317 8498 master-stress-test.cc:579] Killing test thread 1/10
W20260926 01:57:26.216630 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 6: this message will repeat every 5th retry.
I20260926 01:57:26.452663 12927 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:26.453255 12927 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:26.455262 12927 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:26.457180 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:57:26.457753 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:26.458046 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:26.458700 12927 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:26.640960 12927 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:26.641495 12927 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:26.644800 12927 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:26.646863 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:57:26.647576 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:26.647931 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:26.648778 12927 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:26.863888 12933 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:26.864349 12933 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:26.865981 12933 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:26.871317 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:26.871809 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:26.872166 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:26.872771 12933 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:57:27.400667 12941 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:27.401121 12941 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:27.402905 12941 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:27.408226 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:27.408643 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:27.408973 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:27.409499 12941 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:57:28.855842 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:28.856289 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:29.254305 12962 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:29.254904 12962 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:29.256582 12962 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:29.261061 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:29.261523 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:29.261859 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:29.262418 12962 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:57:31.428692 12980 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:31.429141 12980 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:31.430799 12980 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:31.432919 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260926 01:57:31.435988 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:31.436456 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:31.436836 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:31.437389 12980 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:57:32.770913 12993 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:32.771384 12993 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:32.773705 12993 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:32.775508 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:57:32.776109 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:32.776348 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:32.776839 12993 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:33.156951 12998 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:33.157583 12998 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:33.159499 12998 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:33.165627 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:33.166266 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:33.166646 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:33.167279 12998 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:57:33.862486 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:33.862974 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:34.147356 13012 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:34.147760 13012 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:34.149336 13012 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:34.153892 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:34.154340 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:34.154724 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:34.155293 13012 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:57:36.860513 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 11: this message will repeat every 5th retry.
I20260926 01:57:38.868567 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:38.868943 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:39.307324 13056 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:39.308055 13056 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:39.310292 13056 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:39.311884 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:57:39.312397 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:39.312673 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:39.313215 13056 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:40.394755 13066 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:40.395207 13066 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:40.396812 13066 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:40.399441 12782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:40.399441 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260926 01:57:40.402738 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:57:40.403142 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:57:40.403537 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:40.404129 13066 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:57:43.418318 12358 debug-util.cc:398] Leaking SignalData structure 0x7b080003eb20 after lost signal to thread 12228
W20260926 01:57:43.419735 12358 debug-util.cc:398] Leaking SignalData structure 0x7b080003e8e0 after lost signal to thread 12361
I20260926 01:57:43.876235 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:43.876763 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:44.219808 13111 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:57:44.220793 13111 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:57:44.230635 13111 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:57:44.254333 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:57:44.255106 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:57:44.255586 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:57:44.256381 13111 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:57:45.243348 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000d6660 after lost signal to thread 12083
W20260926 01:57:45.244385 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000833a0 after lost signal to thread 12215
I20260926 01:57:46.301160 13134 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:46.301941 13134 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:46.304452 13134 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 01:57:46.305076 13134 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:46.305513 13134 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:57:46.307380 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:46.307505 13134 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:46.310053 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:57:46.310792 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:46.311151 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:46.311844 13134 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:57:46.312237 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:46.312633 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:46.313246 13134 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:57:47.260244 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 16: this message will repeat every 5th retry.
I20260926 01:57:48.882660 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:48.883046 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:57:51.759966 13205 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:51.760440 13205 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:51.762573 13205 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:51.764320 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:57:51.764874 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:57:51.765131 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:51.765677 13205 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:51.924916 13205 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:57:51.925395 13205 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:57:51.927122 13205 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:57:51.928751 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:57:51.929224 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:57:51.929477 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:57:51.929986 13205 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:57:53.890645 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:53.891259 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:57:57.620180 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 21: this message will repeat every 5th retry.
I20260926 01:57:58.900019 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:57:58.900436 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:58:01.007151 13324 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:58:01.007707 13324 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:58:01.009919 13324 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:58:01.012266 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20260926 01:58:01.016579 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:58:01.017179 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:58:01.017544 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:58:01.018205 13324 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:58:03.907694 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:03.908218 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:58:04.532757 13372 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:58:04.533442 13372 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:58:04.535487 13372 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:58:04.540737 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:58:04.541271 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:58:04.541584 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:58:04.542150 13372 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:58:07.883538 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 26: this message will repeat every 5th retry.
I20260926 01:58:08.914187 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:08.914687 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:58:09.673915 13430 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:09.674603 13430 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:09.676925 13430 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:09.678915 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:58:09.679495 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:58:09.679750 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:09.680341 13430 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:58:10.416059 13444 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:10.416591 13444 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:10.418367 13444 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:10.420121 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:58:10.420627 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:58:10.420864 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:10.421380 13444 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:58:13.354974 12217 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (110 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:58:13.667416 12363 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (111 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:58:13.768289 12072 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (144 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:58:13.923659 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:13.924326 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:58:17.407110 13531 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:17.407667 13531 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:17.409550 13531 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:17.411212 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:58:17.411743 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:58:17.412022 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:17.412595 13531 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:58:17.905542 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 31: this message will repeat every 5th retry.
I20260926 01:58:18.931962 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:18.932384 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:58:20.546154 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc820 after lost signal to thread 12748
W20260926 01:58:20.546972 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000b19a0 after lost signal to thread 12836
I20260926 01:58:21.127104 13583 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:58:21.127792 13583 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:58:21.130353 13583 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:58:21.133173 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260926 01:58:21.137051 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:58:21.137554 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:58:21.138016 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:58:21.138885 13583 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:58:23.966748 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:23.967226 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:58:28.084820 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 36: this message will repeat every 5th retry.
I20260926 01:58:28.972865 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:28.973259 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:58:29.471735 13686 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:29.472326 13686 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:29.475171 13686 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:29.477046 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:58:29.477623 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:58:29.477897 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:29.478523 13686 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:58:29.550626 13686 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:29.551074 13686 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:29.552780 13686 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:29.554510 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:58:29.555065 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:58:29.555418 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:29.556108 13686 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:58:33.836473 13743 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:58:33.837185 13743 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:58:33.839313 13743 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:58:33.842556 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 01:58:33.845954 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:58:33.846437 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:58:33.846784 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:58:33.847375 13743 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:58:33.978749 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:33.979199 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:58:35.990259 12067 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5b60 after lost signal to thread 11937
W20260926 01:58:35.992200 12067 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2be0 after lost signal to thread 12070
W20260926 01:58:38.919426 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000706c0 after lost signal to thread 12748
W20260926 01:58:38.920053 12833 debug-util.cc:398] Leaking SignalData structure 0x7b0800070620 after lost signal to thread 12836
I20260926 01:58:39.021803 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:39.022185 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:58:39.166389 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 41: this message will repeat every 5th retry.
I20260926 01:58:43.679011 13873 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:43.679802 13873 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:43.682340 13873 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:43.684610 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:58:43.685132 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:58:43.685405 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:43.686010 13873 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:58:44.028679 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:44.029167 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:58:46.600687 13902 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:46.601308 13902 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:46.603821 13902 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:46.605464 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:58:46.606001 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:58:46.606237 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:46.606813 13902 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:58:49.034761 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:49.035203 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:58:49.895076 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 46: this message will repeat every 5th retry.
I20260926 01:58:50.926818 13948 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:58:50.927353 13948 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:58:50.929471 13948 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:58:50.932582 12782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 01:58:50.937203 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:58:50.938156 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:58:50.938421 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:58:50.939024 13948 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:58:54.042030 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:54.042531 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:58:57.458041 14033 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:58:57.458663 14033 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:58:57.460626 14033 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:58:57.463790 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 01:58:57.467427 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:58:57.468048 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:58:57.468323 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:58:57.468873 14033 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:58:57.991349 14041 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:58:57.991950 14041 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:58:57.993824 14041 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:58:57.995609 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:58:57.996165 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:58:57.996425 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:58:57.996997 14041 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:58:59.048815 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:58:59.049232 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:58:59.994699 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 51: this message will repeat every 5th retry.
I20260926 01:59:01.130722 14068 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:01.131196 14068 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:01.133044 14068 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:01.134765 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:59:01.135433 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:59:01.135807 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:01.136847 14068 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:01.535254 14068 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:01.535768 14068 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 01:59:01.544277 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:59:01.550340 14068 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:01.556020 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:59:01.556483 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:01.558667 14068 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:02.425390 14097 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:59:02.426023 14097 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:59:02.428001 14097 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:59:02.434624 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:59:02.435549 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:59:02.435967 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:59:02.436744 14097 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:59:02.568410 12358 debug-util.cc:398] Leaking SignalData structure 0x7b0800040e20 after lost signal to thread 12228
W20260926 01:59:02.569486 12358 debug-util.cc:398] Leaking SignalData structure 0x7b0800040540 after lost signal to thread 12361
I20260926 01:59:04.055931 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:04.056406 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:59:08.781556 14163 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:08.782197 14163 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:08.784552 14163 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:08.786478 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:59:08.787088 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:59:08.787360 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:08.787928 14163 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:09.063905 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:09.064339 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:59:09.850421 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 56: this message will repeat every 5th retry.
I20260926 01:59:10.458142 14194 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:59:10.458626 14194 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:59:10.460508 14194 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:59:10.463013 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260926 01:59:10.466544 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:59:10.467993 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:59:10.468287 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:59:10.468892 14194 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 01:59:13.649806 12217 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (230 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:59:13.873237 12212 debug-util.cc:398] Leaking SignalData structure 0x7b0800044280 after lost signal to thread 12083
W20260926 01:59:13.874272 12212 debug-util.cc:398] Leaking SignalData structure 0x7b0800040ba0 after lost signal to thread 12215
W20260926 01:59:14.004904 12072 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (296 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:59:14.071650 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:14.072165 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:59:14.099457 12362 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (216 consecutive failures): Network error: Failed to ping master at 127.8.76.188:36403: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:59:15.489063 14253 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:15.489529 14253 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:15.491272 14253 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:15.493033 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:59:15.493584 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:59:15.493860 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:15.494410 14253 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:19.077654 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:19.078032 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:59:19.634410 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 61: this message will repeat every 5th retry.
I20260926 01:59:20.185442 14315 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:59:20.186007 14315 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:59:20.187947 14315 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:59:20.190372 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 01:59:20.193629 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:59:20.194036 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:59:20.194377 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:59:20.195020 14315 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:59:22.313668 14343 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:22.314258 14343 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:22.316457 14343 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:22.318156 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:59:22.318813 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:59:22.319095 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:22.319773 14343 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:23.812948 12789 maintenance_manager.cc:419] P d9760316a8f74e829b4c7884b875b70e: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033361
I20260926 01:59:23.894639 12788 maintenance_manager.cc:643] P d9760316a8f74e829b4c7884b875b70e: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.080s user 0.043s sys 0.016s Metrics: {"bytes_written":11072,"cfile_init":1,"drs_written":1,"lbm_read_time_us":225,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2292,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":19,"thread_start_us":485,"threads_started":1,"wal-append.queue_time_us":801}
I20260926 01:59:23.897038 12789 maintenance_manager.cc:419] P d9760316a8f74e829b4c7884b875b70e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1051 bytes on disk
I20260926 01:59:23.898730 12788 maintenance_manager.cc:643] P d9760316a8f74e829b4c7884b875b70e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":238,"lbm_reads_lt_1ms":4}
I20260926 01:59:24.085175 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:24.085641 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:59:24.813524 14372 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:24.813971 14372 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:24.816080 14372 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:24.817598 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:59:24.818200 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:59:24.818543 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:24.819113 14372 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:59:28.025205 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000d50e0 after lost signal to thread 12083
W20260926 01:59:28.026283 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000c4a20 after lost signal to thread 12215
I20260926 01:59:29.091573 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:29.091995 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:59:29.216059 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 66: this message will repeat every 5th retry.
I20260926 01:59:29.798497 14422 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:29.799103 14422 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:29.800930 14422 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:29.802673 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:59:29.803201 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:59:29.803459 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:29.804056 14422 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:32.748677 14466 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:32.749171 14466 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:32.750980 14466 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:32.752336 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:59:32.752875 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:59:32.753131 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:32.753674 14466 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:34.099115 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:34.099635 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:59:34.608913 12067 debug-util.cc:398] Leaking SignalData structure 0x7b08000e15c0 after lost signal to thread 11937
W20260926 01:59:34.610220 12067 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2260 after lost signal to thread 12070
W20260926 01:59:35.897437 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7540 after lost signal to thread 12748
W20260926 01:59:35.898096 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6cc0 after lost signal to thread 12836
I20260926 01:59:36.475317 14538 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:59:36.475881 14538 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:59:36.478135 14538 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:59:36.480495 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 01:59:36.483502 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 01:59:36.483956 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 01:59:36.484270 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:59:36.484896 14538 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:59:39.186188 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:39.186667 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:59:40.044390 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 71: this message will repeat every 5th retry.
I20260926 01:59:44.193583 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:44.194140 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:59:49.023396 14733 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:49.024008 14733 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:49.026559 14733 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:49.028319 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:59:49.028810 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:59:49.029052 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:49.029611 14733 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:49.185721 14733 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:49.186236 14733 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:49.187961 14733 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:49.189556 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:59:49.190033 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:59:49.190267 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:49.190805 14733 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:49.199644 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:49.200052 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 01:59:50.306053 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 76: this message will repeat every 5th retry.
I20260926 01:59:50.504338 14771 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 01:59:50.505100 14771 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 01:59:50.508090 14771 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 01:59:50.510375 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 01:59:50.514848 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 01:59:50.515666 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 01:59:50.516053 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 01:59:50.516795 14771 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 01:59:54.206426 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:54.206943 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:59:56.102005 14851 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:56.102717 14851 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:56.104692 14851 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:56.106537 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 01:59:56.107084 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 01:59:56.107368 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:56.107941 14851 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 01:59:59.212606 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 01:59:59.212993 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 01:59:59.600548 14923 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 01:59:59.601112 14923 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 01:59:59.603055 14923 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 01:59:59.604877 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 01:59:59.605490 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 01:59:59.605757 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 01:59:59.606341 14923 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 01:59:59.847725 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 81: this message will repeat every 5th retry.
I20260926 02:00:04.219933 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:04.220431 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:09.127475 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 86: this message will repeat every 5th retry.
I20260926 02:00:09.225970 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:09.226502 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:00:09.410395 15079 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:00:09.411020 15079 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:00:09.413384 15079 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:00:09.415342 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:00:09.420327 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:00:09.421180 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:00:09.421474 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:00:09.422080 15079 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:00:13.546535 15147 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:00:13.547143 15147 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:00:13.549500 15147 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:00:13.551182 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:00:13.551817 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:00:13.552095 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:00:13.552651 15147 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 02:00:13.941301 12217 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (350 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:00:14.173738 12071 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (426 consecutive failures): Network error: Failed to ping master at 127.8.76.188:36403: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:00:14.233546 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:14.234100 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:14.528218 12362 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (336 consecutive failures): Network error: Failed to ping master at 127.8.76.188:36403: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:00:15.075073 15161 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:00:15.075547 15161 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:00:15.077207 15161 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:00:15.079085 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:00:15.079577 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:00:15.079804 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:00:15.080325 15161 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:00:19.239609 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:19.240012 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:19.298969 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 91: this message will repeat every 5th retry.
W20260926 02:00:19.672037 12358 debug-util.cc:398] Leaking SignalData structure 0x7b08000c31a0 after lost signal to thread 12228
W20260926 02:00:19.673080 12358 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9720 after lost signal to thread 12361
W20260926 02:00:21.540000 12067 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf200 after lost signal to thread 11937
W20260926 02:00:21.541034 12067 debug-util.cc:398] Leaking SignalData structure 0x7b080003d080 after lost signal to thread 12070
I20260926 02:00:24.247144 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:24.247619 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:25.963760 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd180 after lost signal to thread 12083
W20260926 02:00:25.964778 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1240 after lost signal to thread 12215
I20260926 02:00:28.628086 15389 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:00:28.628978 15389 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:00:28.638234 15389 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:00:28.648403 12782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:00:28.661285 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:00:28.661963 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 02:00:28.662269 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:00:28.662931 15389 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 02:00:28.761636 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 96: this message will repeat every 5th retry.
I20260926 02:00:29.253310 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:29.253684 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:00:29.454361 15400 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:00:29.455008 15400 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:00:29.457424 15400 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:00:29.459239 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:00:29.459858 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:00:29.460132 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:00:29.460739 15400 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:00:30.370043 15419 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:00:30.370604 15419 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:00:30.372504 15419 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:00:30.377439 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:00:30.377858 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:00:30.378237 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:00:30.378906 15419 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 02:00:30.547938 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8400 after lost signal to thread 12083
W20260926 02:00:30.549193 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000c70a0 after lost signal to thread 12215
I20260926 02:00:31.275110 15430 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:00:31.275599 15430 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:00:31.277421 15430 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:00:31.279234 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:00:31.279757 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:00:31.279999 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:00:31.280498 15430 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:00:34.260182 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:34.260660 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:38.769627 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 101: this message will repeat every 5th retry.
I20260926 02:00:39.266510 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:39.266943 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:00:44.273931 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:44.274565 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:00:46.547838 15612 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:00:46.548547 15612 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:00:46.550945 15612 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:00:46.553361 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20260926 02:00:46.553468 15607 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:00:46.554319 15607 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
W20260926 02:00:46.556492 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:00:46.556985 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 02:00:46.557329 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:00:46.557425 15607 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
I20260926 02:00:46.558060 15612 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260926 02:00:46.559309 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:00:46.559924 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:00:46.560186 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:00:46.560765 15607 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:00:46.760808 15607 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:00:46.761267 15607 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:00:46.763252 15607 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:00:46.764843 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:00:46.765348 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:00:46.765607 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:00:46.766094 15607 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:00:49.280409 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:49.280792 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:49.599470 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 106: this message will repeat every 5th retry.
I20260926 02:00:53.782335 15714 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:00:53.782922 15714 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:00:53.784586 15714 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:00:53.786321 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:00:53.786868 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:00:53.787101 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:00:53.787588 15714 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:00:54.323395 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:54.323845 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:54.936326 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000bbde0 after lost signal to thread 12748
W20260926 02:00:54.936996 12833 debug-util.cc:398] Leaking SignalData structure 0x7b0800070380 after lost signal to thread 12836
I20260926 02:00:59.335906 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:00:59.336272 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:00:59.519922 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 111: this message will repeat every 5th retry.
I20260926 02:01:01.237142 15832 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:01:01.237661 15832 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:01:01.239385 15832 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:01:01.241387 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:01:01.244647 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:01:01.245250 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:01:01.245548 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:01:01.246171 15832 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:01:01.846128 15842 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:01.846861 15842 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:01.850584 15842 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:01.852063 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:01:01.852613 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:01:01.852903 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:01.853468 15842 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:03.992334 15885 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:01:03.993050 15885 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:01:03.996117 15885 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:01:04.002818 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:01:04.003470 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:01:04.003777 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:01:04.004355 15885 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:01:04.341696 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:04.342120 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:01:05.207208 12067 debug-util.cc:398] Leaking SignalData structure 0x7b08000db280 after lost signal to thread 11937
W20260926 02:01:05.208294 12067 debug-util.cc:398] Leaking SignalData structure 0x7b0800040d60 after lost signal to thread 12070
I20260926 02:01:09.347755 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:09.348124 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:01:09.433611 15974 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:09.434259 15974 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:09.436264 15974 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:09.438367 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:01:09.438966 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:01:09.439275 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:09.440001 15974 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 02:01:09.543946 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 116: this message will repeat every 5th retry.
I20260926 02:01:10.360463 16006 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:10.360968 16006 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:10.362812 16006 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:10.364382 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:01:10.364928 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:01:10.365206 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:10.365712 16006 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 02:01:14.314814 12216 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (456 consecutive failures): Network error: Failed to ping master at 127.8.76.188:36403: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:01:14.349841 12072 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (595 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 02:01:14.354786 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:14.355326 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:01:14.946827 12363 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (471 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 02:01:16.081133 16109 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:16.082028 16109 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:16.084626 16109 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:16.086442 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:01:16.087070 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:01:16.087356 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:16.087975 16109 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:17.241679 16138 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:01:17.242302 16138 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:01:17.244370 16138 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:01:17.246691 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260926 02:01:17.250664 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:01:17.252264 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:01:17.252558 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:01:17.253144 16138 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:01:19.360909 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:19.361304 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:01:19.629401 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 121: this message will repeat every 5th retry.
I20260926 02:01:24.368261 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:24.368723 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:01:26.240512 16320 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:01:26.241070 16320 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:01:26.242864 16320 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:01:26.245374 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:01:26.248682 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:01:26.249341 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:01:26.249639 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:01:26.250278 16320 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:01:27.247339 16325 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:27.248139 16325 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:27.251516 16325 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:27.253213 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:01:27.253858 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:01:27.254117 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:27.254788 16325 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:28.363217 16355 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:28.363791 16355 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:28.365612 16355 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:28.367152 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:01:28.367687 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:01:28.367956 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:28.368476 16355 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:29.374411 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:29.374818 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:01:29.408730 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 126: this message will repeat every 5th retry.
I20260926 02:01:34.381093 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:34.381560 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:01:36.739857 16529 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:01:36.740378 16529 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:01:36.742163 16529 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:01:36.744930 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:01:36.748235 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:01:36.748749 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:01:36.749034 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:01:36.749568 16529 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:01:37.145505 16524 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:37.146129 16524 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:37.147984 16524 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:37.149819 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:01:37.150352 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:01:37.150655 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:37.151247 16524 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:37.718389 16549 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:01:37.718895 16549 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:01:37.720587 16549 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:01:37.726667 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:01:37.727504 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:01:37.727787 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:01:37.728286 16549 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:01:39.387547 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:39.388008 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:01:39.601991 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 131: this message will repeat every 5th retry.
W20260926 02:01:41.736167 12358 debug-util.cc:398] Leaking SignalData structure 0x7b08000db120 after lost signal to thread 12228
W20260926 02:01:41.737702 12358 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9600 after lost signal to thread 12361
I20260926 02:01:43.070176 16652 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:43.070725 16652 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:43.073130 16652 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:43.074746 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:01:43.075409 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:01:43.075691 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:43.076264 16652 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:44.394897 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:44.395347 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:01:48.737182 16728 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:48.737850 16728 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:48.739974 16728 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:48.741672 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:01:48.742264 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:01:48.742587 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:48.743208 16728 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:49.400990 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:49.401351 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:01:50.200379 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 136: this message will repeat every 5th retry.
I20260926 02:01:51.357086 16809 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:01:51.357623 16809 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:01:51.359299 16809 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:01:51.361676 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260926 02:01:51.364468 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:01:51.364885 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 02:01:51.365161 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:01:51.365713 16809 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:01:53.919330 16850 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:01:53.919919 16850 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:01:53.922204 16850 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:01:53.923964 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:01:53.925035 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:01:53.925299 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:01:53.925848 16850 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:01:54.429100 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:54.429569 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:01:54.869693 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000bca20 after lost signal to thread 12748
W20260926 02:01:54.870368 12833 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc840 after lost signal to thread 12836
I20260926 02:01:59.435117 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:01:59.435467 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:02:00.327332 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 141: this message will repeat every 5th retry.
I20260926 02:02:04.442482 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:02:04.442993 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:02:05.533339 17082 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:02:05.533867 17082 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:02:05.535748 17082 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:02:05.538537 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:02:05.541533 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:02:05.542091 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 02:02:05.542371 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:02:05.542981 17082 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:02:05.658797 17073 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:02:05.659396 17073 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:02:05.661621 17073 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:02:05.663339 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:02:05.663965 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:02:05.664320 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:02:05.665011 17073 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:02:08.069598 17130 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:02:08.070274 17130 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:02:08.072211 17130 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:02:08.073961 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:02:08.074599 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:02:08.074863 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:02:08.075467 17130 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:02:09.448637 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:02:09.448994 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:02:10.086836 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 146: this message will repeat every 5th retry.
I20260926 02:02:11.513597 17187 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:02:11.514060 17187 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:02:11.515862 17187 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:02:11.517452 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:02:11.518024 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:02:11.518277 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:02:11.518831 17187 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:02:13.870792 17241 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:02:13.871409 17241 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:02:13.873238 17241 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:02:13.876250 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:02:13.880829 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:02:13.882439 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
I20260926 02:02:13.882905 12783 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:02:13.883757 17241 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:02:14.455670 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:02:14.456133 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:02:14.517268 12072 heartbeater.cc:646] Failed to heartbeat to 127.8.76.189:34661 (742 consecutive failures): Network error: Failed to ping master at 127.8.76.189:34661: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:02:14.749965 12216 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (576 consecutive failures): Network error: Failed to ping master at 127.8.76.188:36403: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:02:15.400066 12362 heartbeater.cc:646] Failed to heartbeat to 127.8.76.188:36403 (576 consecutive failures): Network error: Failed to ping master at 127.8.76.188:36403: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
W20260926 02:02:19.100868 11959 consensus_peers.cc:597] T 5ab9e06a641740c7bbb1afbe0f4943a0 P 5d58342c1ed14ef9a6815b9b5a219b56 -> Peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Couldn't send request to peer 1b877862e4a543229352921130f57068. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab9e06a641740c7bbb1afbe0f4943a0. This is attempt 151: this message will repeat every 5th retry.
I20260926 02:02:19.461948 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:02:19.462332 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:02:20.659143 12212 debug-util.cc:398] Leaking SignalData structure 0x7b08000bca60 after lost signal to thread 12083
W20260926 02:02:20.660207 12212 debug-util.cc:398] Leaking SignalData structure 0x7b0800070800 after lost signal to thread 12215
I20260926 02:02:21.233325 17376 raft_consensus.cc:493] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:02:21.234038 17376 raft_consensus.cc:515] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:02:21.236903 17376 leader_election.cc:290] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:02:21.238801 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
W20260926 02:02:21.239598 11959 leader_election.cc:343] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 2ea4233817144476b1397f95eb68c5da
I20260926 02:02:21.239944 11959 leader_election.cc:304] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:02:21.240540 17376 raft_consensus.cc:2749] T 2ea4233817144476b1397f95eb68c5da P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:02:23.249238 17417 raft_consensus.cc:493] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 596cdfef9f194805948e35ffe299ed65)
I20260926 02:02:23.249837 17417 raft_consensus.cc:515] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9760316a8f74e829b4c7884b875b70e" member_type: VOTER last_known_addr { host: "127.8.76.190" port: 41989 } } peers { permanent_uuid: "6819a8c0ba75415c9cf089983ff5c191" member_type: VOTER last_known_addr { host: "127.8.76.189" port: 34661 } } peers { permanent_uuid: "596cdfef9f194805948e35ffe299ed65" member_type: VOTER last_known_addr { host: "127.8.76.188" port: 36403 } }
I20260926 02:02:23.251806 17417 leader_election.cc:290] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661), 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403)
W20260926 02:02:23.254813 12783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260926 02:02:23.258313 12783 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6819a8c0ba75415c9cf089983ff5c191 (127.8.76.189:34661): Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111)
W20260926 02:02:23.259091 12782 leader_election.cc:336] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 596cdfef9f194805948e35ffe299ed65 (127.8.76.188:36403): Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection refused (error 111)
I20260926 02:02:23.259438 12782 leader_election.cc:304] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9760316a8f74e829b4c7884b875b70e; no voters: 596cdfef9f194805948e35ffe299ed65, 6819a8c0ba75415c9cf089983ff5c191
I20260926 02:02:23.260115 17417 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d9760316a8f74e829b4c7884b875b70e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260926 02:02:24.469190 12741 master_runner.cc:187] Error getting master registration for 127.8.76.189:34661: Network error: Client connection negotiation failed: client connection to 127.8.76.189:34661: connect: Connection refused (error 111),
I20260926 02:02:24.469658 12741 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:02:24.878000 12789 maintenance_manager.cc:419] P d9760316a8f74e829b4c7884b875b70e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1051 bytes on disk
I20260926 02:02:24.892961 12788 maintenance_manager.cc:643] P d9760316a8f74e829b4c7884b875b70e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"dirs.queue_time_us":2181,"dirs.run_cpu_time_us":325,"dirs.run_wall_time_us":327,"thread_start_us":2456,"threads_started":2,"wal-append.queue_time_us":1028}
I20260926 02:02:25.095635 17436 raft_consensus.cc:493] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:02:25.096210 17436 raft_consensus.cc:515] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d58342c1ed14ef9a6815b9b5a219b56" member_type: VOTER last_known_addr { host: "127.8.76.129" port: 41369 } } peers { permanent_uuid: "1b877862e4a543229352921130f57068" member_type: VOTER last_known_addr { host: "127.8.76.130" port: 34281 } } peers { permanent_uuid: "495ab022dafc493395ae0e21d9a77ee3" member_type: VOTER last_known_addr { host: "127.8.76.131" port: 46707 } }
I20260926 02:02:25.097946 17436 leader_election.cc:290] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b877862e4a543229352921130f57068 (127.8.76.130:34281), 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707)
W20260926 02:02:25.099742 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b877862e4a543229352921130f57068 (127.8.76.130:34281): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
W20260926 02:02:25.100250 11959 leader_election.cc:343] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 495ab022dafc493395ae0e21d9a77ee3 (127.8.76.131:46707): Not found: Tablet not found: 6ac6c587687c4d868586da53310dfa5b
I20260926 02:02:25.100481 11959 leader_election.cc:304] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d58342c1ed14ef9a6815b9b5a219b56; no voters: 1b877862e4a543229352921130f57068, 495ab022dafc493395ae0e21d9a77ee3
I20260926 02:02:25.100979 17436 raft_consensus.cc:2749] T 6ac6c587687c4d868586da53310dfa5b P 5d58342c1ed14ef9a6815b9b5a219b56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 02:02:25.184912 12372 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection reset by peer (error 104)
W20260926 02:02:25.185576 12372 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection reset by peer (error 104)
W20260926 02:02:25.185922 12372 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection reset by peer (error 104)
W20260926 02:02:25.186137 12372 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection reset by peer (error 104)
F20260926 02:02:25.197547 12386 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection reset by peer (error 104)
*** Check failure stack trace: ***
F20260926 02:02:25.201061 12388 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.76.190:41989,127.8.76.189:34661,127.8.76.188:36403, num_attempts: 768) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.76.188:36403: connect: Connection reset by peer (error 104)
*** Aborted at 1790388145 (unix time) try "date -d @1790388145" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800002132) received by PID 8498 (TID 0x7f80894f8700) from PID 8498; stack trace: ***
@ 0x2b0620 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b2b03 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f809ddf5980 (unknown) at ??:0
@ 0x7f8098f9fe87 gsignal at ??:0
@ 0x7f8098fa17f1 abort at ??:0
@ 0x2b6416 __interceptor_abort at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f809c03dd74 google::LogMessage::SendToLog() at ??:0
@ 0x7f809c03e920 google::LogMessage::Flush() at ??:0
@ 0x7f809c043a5b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35ec5e kudu::MasterStressTest::DeleteTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38bc3e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38baf1 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38b23b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2b04e0 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f809ddea6db start_thread at ??:0
@ 0x7f809908261f clone at ??:0